-
-
Notifications
You must be signed in to change notification settings - Fork 328
Support all indexing variants #1917
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Thanks for this effort! I will give it a closer look later today. regarding these two concerns:
I don't have a lot of experience with either of these features from v2, so I am probably not the best judge of how we want this to look. Are there any big time 0-dimensional array or object dtype users we can ping to have them look it over?
Similarly, I never use |
Regarding As @akshaysubr point out in #1751 (comment), it might be diffecult to support an output-by-caller policy so I am leaning to remove the |
While I agree that providing an out kwarg is pointless if zarr-python is doing copies anyways, we have it in the existing API. I wonder if we should continue to provide the out arg but issue a (deprecation) warning? |
Good point, a deprecation warning is a good idea :) |
#1874 recently revealed that there are definitely users of 0-dimensional arrays. (Also python objects dtypes.) |
src/zarr/indexing.py
Outdated
BlockSelection = BlockSelector | tuple[BlockSelector, ...] | ||
BlockSelectionNormalized = tuple[BlockSelector, ...] | ||
MaskSelection = npt.NDArray[np.bool_] | ||
OrthogonalSelector = int | slice | npt.NDArray[np.intp | np.bool_] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should look for a way to simplify this as much as we can.OrthogonalSelector
vs OrthogonalSelection
, where the latter contains the former, is a recipe for confusion.
CoordinateSelection = npt.NDArray[np.intp] | ||
BlockSelector = int | slice | ||
BlockSelection = BlockSelector | tuple[BlockSelector, ...] | ||
BlockSelectionNormalized = tuple[BlockSelector, ...] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think when we use a type alias like BlockSelectionNormalized
for tuple[BlockSelector,...]
we lose some readability. To know that BlockSelectionNormalized
is tuple[int | slice, ...]
I have to do 2 lookups, and we aren't even saving lines of code because the type aliases are longer than the types. Was there a particular problem caused by the plain type annotations?
MaskSelection = npt.NDArray[np.bool_] | ||
OrthogonalSelector = int | slice | npt.NDArray[np.intp | np.bool_] | ||
OrthogonalSelection = OrthogonalSelector | tuple[OrthogonalSelector, ...] | ||
OrthogonalSelectionNormalized = tuple[OrthogonalSelector, ...] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From the name OrthogonalSelectionNormalized
I would expect that this type takes OrthogonalSelection
, but it takes OrthogonalSelecTOR
, which is a bit surprising / confusing
I got the types to a state where mypy doesn't complain anymore. I know that the types are far from ideal. However, it will take more work to iron that out given the variety of indexing methods that zarr-python supports and I don't want to hold this off from the alpha release. |
This is great, thanks @normanrz! Agreed that we can sort out the types in a later effort. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking great @normanrz -- a few comments to help tidy this up.
In this PR I ported over all indexing variants from the v2 codebase into v3. The
Array
class exposes the sames methods as in the v2 code base (e.g.get_basic_selection
,set_orthogonal_selection
,oindex[...]
) whereas theAsyncArray
only has_get_selection(indexer, ...)
and_set_selection(indexer, ...)
.The current status of this PR is: the tests are green but typing still causes some headaches.
There are a few breaking changes of the v3 code that we should discuss:
out
kwarg for getitem doesn't work and can be complicated to implement with the new NDBuffer abstractionPlease let me know your thoughts on these limitations @jhamman @d-v-b.
TODO: