STAC to handle storage implementation details and access parameters

[emmanuelmathot]

Context

During the recent STAC+Zarr Community Sprint in Rome and in PR discussions (radiantearth/stac-best-practices#29), we've encountered fundamental questions about STAC's role in describing storage implementation details, particularly for complex stores like Zarr.

This discussion seeks community input on where STAC should draw boundaries between data description and implementation details.

The Challenge

As we integrate more complex data formats (Zarr, virtual stores, cloud-optimized formats), we're facing decisions about what metadata belongs in STAC:

• STAC traditionally describes what data is, not how to access it optimally
• Users increasingly need implementation hints for efficient data access
• Some metadata exists both in the data format (e.g., Zarr's consolidated metadata) and might be replicated in STAC

Specific Examples from Recent Work

1. Chunk/Block Size Information

   • Should STAC include chunk sizes when they're already in Zarr metadata?
   • Use case: Libraries like xarray may benefit from knowing chunk size before opening files. Browser direct access may also need to know if the access will be possible with not too big chunks.
   • Related: Add missing metadata for STACTA driver stac-extensions/raster#49 proposes adding block_size. Not sure this belongs to raster as it invloves more dimensions than just the x and y.

2. Store References in Links

   • Should we use links (not just assets) to reference data stores?
   • Example: rel: "store" linking to Zarr store root
   • This breaks the traditional pattern of links being for relationships, not data

3. Abstract Store Interface Information

   • Should STAC specify more than store types (fsspec, S3, Azure, virtual)?
   • Where do access optimization parameters belong?

Questions for the Community

1. In the scope of implementation hints for performance optimization, should we have a place for storage description?

2. When is it acceptable to duplicate metadata that exists in the data format itself?

3. Should implementation details go in:

   • Existing extensions (e.g., storage, raster)?
   • New abstract store extensions that would then contain information about the implmentation (e.g., zarr, fsspec, chunk_size, etc..)?

4. What implementation details do you find essential when working with complex data formats?

Related Discussions

• Zarr Extension Discussion Zarr Extension? #1222 and zarr extension
• Zarr to STAC best practises Add best practices for STAC Zarr stac-best-practices#29
• VirtualiZarr, Virtual assets and Reference Files
• Collection-level Assets for Large Stores

[Scartography]

• Clear href templating solution for accessing groups (nested) in the .zarr should also be adressed at (STAC) metadata level.
• It should always be clear by asset or asset group description, item properties, with what tools and by what structure as well as at what location the datasets can be accessed
• If a asset is not available this should be specified by product extention (as flag)

[m-mohr]

A couple of thoughts:

1. Chunk/Block Size Information: We can make such metadata available, but ideally it works across file formats. We should not have file format specific properties all over the place. All properties need to have a proven usecase within STAC.
2. Store References in Links: Why is an asset not suitable?
3. Abstract Store Interface Information: Feels like this is mixing a couple of things, e.g. S3 is a protocol, fsspec (afaik) is a Python implementation. I'm not a fan of providing information that is only useful in a specific programming language, it should be abstracted so that it is more generally applicable.