Skip to content

hardcode ZENODO_FILES_DATA in WorldCerealConfidences #206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 24, 2025

Conversation

gabrieltseng
Copy link
Contributor

This prevents the following:

requests.exceptions.HTTPError: 429 Client Error: TOO MANY REQUESTS for url: https://zenodo.org/api/deposit/depositions/7875105/files

Since any changes to the files changes the ZENODO_RECORD_ID, I think this is safe to do.

@gabrieltseng gabrieltseng changed the title hardcode ZENODO_FILES_DATA hardcode ZENODO_FILES_DATA in WorldCerealConfidences Jul 11, 2025
Copy link
Collaborator

@favyen2 favyen2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other data sources that require something similar, we cache the data to a file and then read it in the parallel workers so that as long as --no-use-initial-job is NOT passed then it will save the file on the initial job and then afterward they workers will just read from the file.

For example, the first call to LocalFiles.list_items writes a summary.json showing the bounds of all of the raster files and then successive calls can read that instead of opening a lot of GeoTIFFs to get the metadata.

I think it is a bit different here since there are already a fixed set of files you are expecting, so I think it is fine to hardcode if you prefer, but I would cut it down to just the minimal amount of data we need (filename and download link) instead of keeping the entire response.

Copy link
Collaborator

@favyen2 favyen2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!

@gabrieltseng gabrieltseng merged commit b7d9c10 into master Jul 24, 2025
4 checks passed
@gabrieltseng gabrieltseng deleted the gabi/wc-zenodo-fix branch July 24, 2025 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants