trouble saving zarr store directly to google cloud

[naomi-henderson]

This issue is related to #681, which is now closed. I have been trying to get this to work reliably for awhile - hopefully someone can tell me what I am missing.

I open a perfectly normal zarr dataset (my credentials are already cached):

import gcsfs
import xarray as xr
gcs = gcsfs.GCSFileSystem(token='cache', access='read_write')
zstore = 'gs://cmip6/AerChemMIP/BCC/BCC-ESM1/ssp370/r1i1p1f1/Amon/pr/gn/'
ds = xr.open_zarr(zstore,consolidated=True)

and ds looks perfectly normal:

<xarray.Dataset>
Dimensions:    (bnds: 2, lat: 64, lon: 128, time: 492)
Coordinates:
  * lat        (lat) float64 -87.86 -85.1 -82.31 -79.53 ... 82.31 85.1 87.86
    lat_bnds   (lat, bnds) float64 dask.array<chunksize=(64, 2), meta=np.ndarray>
  * lon        (lon) float64 0.0 2.812 5.625 8.438 ... 348.8 351.6 354.4 357.2
    lon_bnds   (lon, bnds) float64 dask.array<chunksize=(128, 2), meta=np.ndarray>
  * time       (time) object 2015-01-16 12:00:00 ... 2055-12-16 12:00:00
    time_bnds  (time, bnds) object dask.array<chunksize=(492, 2), meta=np.ndarray>
...etc

But now I try to write this zarr store directly to GCS:

ds.to_zarr(gcs.get_mapper('gs://cmip6/ztemp9/pr/gn/'),consolidated=True)

Unfortunately, the zarr store it writes is corrupt. For example:

xr.open_zarr(gcs.get_mapper('gs://cmip6/ztemp9/pr/gn/'),consolidated=True)

returns a dataset with no variables and coordinates (but attributes are all fine):

<xarray.Dataset>
Dimensions:  ()
Data variables:
    *empty*
Attributes:
    Conventions:            CF-1.7 CMIP-6.2
 ...etc

Oddly, the zarr store is still about the same size, about 13.8M.

When writing to local disk, to_zarr works fine on this example and does not drop the variable and coordinate info.

I have tried many permutations on this. For example, with and without consolidated=True. Without consolidated=True the store only contains .zarray and .zgroup no subdirectories - so it is even worse.

What am I doing wrong?

[rabernat]

Naomi, thanks for the useful issue.

I have hit this problem before as well. AFAIK, it is related to consolidated=True. If you remove the consolidated option, and re-create a new GCSFileSystem object, it should work.

The root of the problem is related to fsspec / gcsfs caching mechanisms. I think that the directory listing of the path gs://cmip6/ztemp9/pr/gn/ is cached when you first create the mapper object. This cache is not updated even after you write the zarr files. So consolidating the metadata doesn't work.

The problem should probably be raised in the fsspec repo. Perhaps @martindurant can advise us of how to boil it down further.

[naomi-henderson]

Ah, yes - if I re-create the GCSFileSystem before each call, the non-consolidated now works. At least that narrows down my confusion a bit. I wasn't actually using fsspec, just gcsfc - but agree that @martindurant is our best hope since it would be hard for me to explain to the gcsfc folks

[martindurant]

I am trying to model why this happens. My thought is that maybe the files are all written to the store by the workers, but the instance in the client session isn't aware that anything happened, so it retains the version of the directory structure from before, resulting in empty metadata. It would make sense to call invalidate_cache whenever writing happens - that would happen within the xarray code.

This kind of situation has come up a lot. The purpose of caching file listings, of course, is that each call is at least 100ms (from outside google), and may be longer for long listings. Here are some thoughts on a more rigorous solution.

• Dask itself could call invalidate_cache on any fsspec compliant instance when writing might be happening in other processed (for every write-capable method)
• Caching could be made optional or expiring, yet another user-specified parameter. GCS does not provide a checksum of listings to be able to see whether something has changed without downloading the whole new listing.
• Another idea might be to have a "writing context" within dask which would allow for transaction-like behaviour (i.e., success or rollback) and finalisation action; this would take some work to implement, but would be rather useful by itself
• specifically to to_zarr, the call should return back the set of paths it created (like some other write functions do), so we know the information even without querying GCS (which isn't necessarily up to date anyway).

[rabernat]

Thanks @martindurant for the brainstorming!

call invalidate_cache whenever writing happens - that would happen within the xarray code.

The problem is that xarray knows nothing about fsspec and its methods. It is just using zarr's API, and the only thing zarr knows about the store is that it is a MutableMapping.

[martindurant]

It is just using zarr's API

Isn't the marshalling of chunks happening in dask? That might be the logical place.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.