Skip to content

fix consolidate_metadata with FSStore #916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 6, 2022
Merged

fix consolidate_metadata with FSStore #916

merged 2 commits into from
Jan 6, 2022

Conversation

jhamman
Copy link
Member

@jhamman jhamman commented Dec 20, 2021

This PR fixes a small bug in the consolidate_metadata convenience function where, when using fsspec compatible store paths, the FSStore was opened as read-only, causing the write of the consolidated metadata key to fail.

Closes #915

cc @martindurant, @freeman-lab

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@martindurant
Copy link
Member

+1 for the idea

I wonder if we should check that store is a string before doing this - might we otherwise end up writing to a store that has been opened explicitly as readonly? I'm not sure if that's a side-effect, maybe this comment can be ignored.

@codecov
Copy link

codecov bot commented Dec 20, 2021

Codecov Report

Merging #916 (fd8ca6f) into master (5f7e2b6) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #916   +/-   ##
=======================================
  Coverage   99.94%   99.94%           
=======================================
  Files          32       32           
  Lines       11216    11219    +3     
=======================================
+ Hits        11210    11213    +3     
  Misses          6        6           
Impacted Files Coverage Δ
zarr/convenience.py 100.00% <100.00%> (ø)
zarr/tests/test_storage.py 100.00% <100.00%> (ø)

@jhamman
Copy link
Member Author

jhamman commented Dec 20, 2021

@martindurant -

I wonder if we should check that store is a string before doing this - might we otherwise end up writing to a store that has been opened explicitly as readonly? I'm not sure if that's a side-effect, maybe this comment can be ignored.

I think this is already take care of in normalize_store_arg:

zarr-python/zarr/storage.py

Lines 112 to 115 in 4fb0f7e

if isinstance(store, str):
mode = mode if clobber else "r"
if "://" in store or "::" in store:
return FSStore(store, mode=mode, **(storage_options or {}))

@joshmoore
Copy link
Member

@jhamman
Copy link
Member Author

jhamman commented Dec 21, 2021

Thanks, @jhamman! This could fix or at least play a role in fixing:

It's quite possible these are related, but I think the relation is, at most, limited to how FSStore is instantiated in various places. The issue (and changes here) are confined to consolidate_metadata and the clobber argument to normalize_store_arg. It may be worth taking a deeper look at the call path for normalize_store_arg to see if there is some relation.

@joshmoore joshmoore merged commit 094eee2 into zarr-developers:master Jan 6, 2022
@jhamman jhamman deleted the fix_consolidate_metadata_read_only_store branch January 6, 2022 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using consolidate_metadata with FSStore result in a read-only store
3 participants