Skip to content

Only catch FileNotFound in mapping get #259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 27, 2020

Conversation

martindurant
Copy link
Member

For discussion whether this is a good idea. Sees to solve zarr's problem, but would be good to test on large datasets in the cloud.

Fixes #255

@martindurant
Copy link
Member Author

Actually, one concrete case for confound would be IsADirectoryError, i.e., we tried to read a path that does exist, but is not a file. It may be reasonable to put a fs.isfile(path) into __getitem__, which also leads to KeyError if False.

@martindurant martindurant merged commit 7e4c486 into fsspec:master Mar 27, 2020
@martindurant martindurant deleted the mapping_error branch March 27, 2020 16:07
@alimanfoo
Copy link

Thanks Martin.

Actually, one concrete case for confound would be IsADirectoryError, i.e., we tried to read a path that does exist, but is not a file. It may be reasonable to put a fs.isfile(path) into __getitem__, which also leads to KeyError if False.

FWIW that is what the zarr DirectoryStore currently does. But for the more general case that could include cloud stores, it might hurt performance to add an isfile check if it involved an extra service call. I'm happy with the PR as it stands.

@openSourcerer9000
Copy link

I'm so confused about this. I'm trying to run this example notebook, sometimes it returns data, and sometimes NaNs, even when querying the same spatiotemporal bounds on different calls! I'm not sure where to ask this, since it may be an S3 thing, but if someone knows the answer: why is this not consistent? How do I ensure my calls receive data?

I'm a Zarr-head, and believe this should be the easiest way to access this CF-compliant data, but how is this usable if you're playing Russian roulette trying to access data and coming up with blanks most of the time?

I'm not sure I'd rather see errors cancelling an entire operation, especially if some data is coming through. But I would like to be able to differentiate between missing data and understand what's hanging up S3 calls.

fsspec v
'2024.3.1'
zarr v
'2.17.1'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Potential bad interaction with zarr and missing chunks
3 participants