Re-implement map_direct functions using general_blockwise #464
Description
The following functions are currently implemented using map_direct, which assumes that the input arrays have been materialized as Zarr arrays so they can be accessed directly, typically in a way that crosses block boundaries.
-
concat -
flip -
groupby_blockwise -
index(i.e.__getitem__) -
map_overlap
It would however be possible to re-implement them using general_blockwise which maps an output block to an arbitrary set of input blocks. The advantage is that the optimizer would then be able to fuse operations to avoid materializing intermediate Zarr arrays, which would potentially be much more efficient.
These are all quite tricky to re-implement (particularly indexing, although the ndindex library would be helpful), so this is more of a long-term aspiration.