Skip to content

Fix .chunks loading lazy backed array data #6721

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 29, 2022

Conversation

dcherian
Copy link
Contributor

@dcherian dcherian commented Jun 24, 2022

@shoyer is there a way to test this?

@shoyer
Copy link
Member

shoyer commented Jun 24, 2022

We have some tests with InaccessibleVariableDataStore for this sort of thing, but I don't know immediately how to hook that into the Zarr backend.

@shoyer
Copy link
Member

shoyer commented Jun 24, 2022

The simplest option would probably be a custom Zarr store that raises an error if you try to look at array data. This could be implemented as a subclass of an existing Zarr store (e.g., the in memory store) that raises an error in __getitem__ is the filename of requests does not start with ..

@dcherian
Copy link
Contributor Author

We have some tests with InaccessibleVariableDataStore for this sort of thing,

great tip! We don't need zarr actually, this was triggering loading of all lazy arrays.

@dcherian dcherian added the plan to merge Final call for comments label Jun 28, 2022
@andersy005
Copy link
Member

Thank you, @dcherian!

@andersy005 andersy005 merged commit 725258e into pydata:main Jun 29, 2022
@dcherian dcherian deleted the fix-lazy-array-chunks branch June 29, 2022 20:08
@rabernat
Copy link
Contributor

Awesome work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
plan to merge Final call for comments
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Accessing chunks on zarr backed xarray seems to load entire array into memory
4 participants