Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make caching smarter by honoring data publication schedule #250

Closed
amotl opened this issue Nov 25, 2020 · 3 comments
Closed

Make caching smarter by honoring data publication schedule #250

amotl opened this issue Nov 25, 2020 · 3 comments

Comments

@amotl
Copy link
Member

amotl commented Nov 25, 2020

Is your feature request related to a problem? Please describe.
When working on #243, we should also think about making the caching layer smarter. With multiple invocations of --period=historical, for example, data should not be downloaded over and over again.

Describe the solution you'd like
Wetterdienst should honor DWD publication times in order to save resources.

For implementing that, the program must know exactly about the publication schedules of each data period. Letting Wetterdienst know about DWD publication times and update definitions also resonates with #225 somehow.

@wetterfrosch
Copy link

Talking about smartness:
When I query eg the past year from historical (2019-01/2019-12), which is within the 10-year timerange of a single file, all historical files are loaded: Currently I count up to three, beginning in 1992. This doesn't seem efficient. Are all of these values loaded into the RAM?

@gutzbenj
Copy link
Member

Dear @wetterfrosch , an update is on its way to manage historical data in this context more smarter.

@gutzbenj
Copy link
Member

gutzbenj commented Dec 4, 2020

This is implemented now by #267

  • this now uses dynamic periods depending on start date and end date
  • high resolution data with multiple files is filtered for only those files which are overlapping with start date and end date

@gutzbenj gutzbenj closed this as completed Dec 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants