Zarr Version

[sjordan29]

I ran Clearwater-Riverine (using ClearWater-data) with zarr3 and got the following: ZarrUserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future. warnings.warn(

If we pin to zarr 2 for now, it eliminates the warning:
template_dataset.to_zarr(self.store_path, mode="w", compute=False, zarr_format=2).

I think this is because the Consolidated Metadata feature is considered experimental for v3 stores: https://zarr.readthedocs.io/en/main/user-guide/consolidated_metadata/.

@aufdenkampe, @ptomasula - thoughts on pinning to zarr_format=2 while this is experimental?

[aufdenkampe]

@sjordan29, thank you for bring up this question!

I think the performance benefits of Zarr 3 are compelling enough to make the effort to upgrade to it for ClearWater.

Since we are writing all these zarr files, can't we eliminate the use of consolidated metadata? Or is there a compelling reason to need it?

My understanding is that Zarr 3 introduced a more consolidated and standardized metadata spec and format, which in most cases eliminates the benefits of consolidated metadata in zarr2. That said, there must be some reason for keeping it around given the lengthly discussion at zarr-developers/zarr-specs#309.

[aufdenkampe]

Here are some blog posts that describe the many benefits of Zarr 3:

• https://earthmover.io/blog/zarr3-release
• https://earthmover.io/blog/xarray-open-zarr-improvements/
  • See the "what about consolidated metadata?" section

And the Zarr 3 Migration Guide:

• https://zarr.readthedocs.io/en/stable/user-guide/v3_migration/

[ptomasula]

@sjordan29, I might be missing something, but can't we use the consolidated=false argument to as (@aufdenkampe suggested) eliminate the use of consolidated metadata? As an aside, we should also explicitly set the zarr_format=3 argument in that same call. I think given there are no legacy files floating around, we should just drop zarr2 support for output.

[sjordan29]

Thanks for looking at this! I hadn't added that consolidated to all the writes, so the error was still showing. I set consolidated=False and zarr_format=3 and added that to my PR #3