Switch tests to use HDF reader instead of kerchunk-based HDF5 reader

[TomNicholas]

Zarr v3 was officially released today, but we still can't just explicitly depend on it because kerchunk doesn't work with zarr v3. There is a compatibility branch of kerchunk, which is sufficient to get kerchunk's readers working again, but development of that branch has stalled due to errors that arise in kerchunk's MultiZarrToZarr code.

We can't just depend directly on this compatibility branch because that's janky AF.

If we try to change our dependency to zarr v3 these things will break:

• kerchunk-based readers (HDF5, netCDF3, and FITS)
• half our integration tests, which round-trip using the HDF5 reader.

Breaking the kerchunk-based readers wouldn't be that bad - we have @sharkinsspatial 's HDF reader as an alternative for opening HDF/netCDF4 files, and the other two are much less complex/commonly used.

Breaking our integration tests isn't really acceptable, so what we should try is swapping all those tests to use Sean's HDF reader instead of the kerchunk-based HDF5 reader. If that works then we can move forward, and the only downside is that we would have to raise a warning in the HDF5, netCDF3, and FITS readers that you need to install the weird compatibility branch of kerchunk. That's better than the current situation, where you need the weird compatibility branch if you want to use zarr v3 (and therefore icechunk!) at all.

cc @norlandrhagen @mpiannucci

[jsignell]

Most of the integration tests in https://github.com/zarr-developers/VirtualiZarr/blob/main/virtualizarr/tests/test_integration.py are parameterized to use both Sean's HDF5 reader and the kerchunk reader.

Do we want to just remove the kerchunk reader from those tests or figure out a pattern for skipping the kerchunk reader run when kerchunk is unavailable?

[TomNicholas]

Most of the integration tests in https://github.com/zarr-developers/VirtualiZarr/blob/main/virtualizarr/tests/test_integration.py are parameterized to use both Sean's HDF5 reader and the kerchunk reader.

Yeah I realised that yesterday too.

Do we want to just remove the kerchunk reader from those tests

See #392 (comment) - the objective is to be able to run the tests without kerchunk even being installed. I think that means that for now we just remove the kerchunk reader from those tests.

[jsignell]

the objective is to be able to run the tests without kerchunk even being installed

Right! But there could be tests that do run if kerchunk is installed and are just skipped otherwise.

Anyways I am going to take this one on now. I think it makes sense to do this one before #376

[TomNicholas]

there could be tests

We want to be able to pin zarr-python>=3.0.0, and run all tests with it in a CI env that only has released versions of packages installed. Currently that completely precludes using Kerchunk at all, so there should not be any tests that even might use Kerchunk.

it makes sense to do this one before #376

I agree, thanks for taking this on.

[jsignell]

there should not be any tests that even might use Kerchunk.

It felt weird to delete tests that exercise functionality. So see what you think of the approach.

[TomNicholas]

It felt weird to delete tests that exercise functionality.

I guess it's fine for us to keep tests that will never be run by the CI (until kerchunk reaches v3 compatibility and can be re-instated).