Skip to content

N5FSStore #793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 45 commits into from
Sep 19, 2021
Merged

N5FSStore #793

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
d26923a
Drop skip_if_nested_chunks from test_storage.py
joshmoore Jun 14, 2021
c06476d
Add failing nested test
joshmoore Jun 14, 2021
ce8b2f0
Make DirectoryStore dimension_separator aware
joshmoore Jun 14, 2021
e183566
Migrate key logic to core rather than storage
joshmoore Jun 14, 2021
449a67f
Fix linting in new test
joshmoore Jun 14, 2021
10c874e
Merge 'origin/master' into fix-dstore
joshmoore Jun 16, 2021
2e4f4d7
Extend the test suite for dim_sep
joshmoore Jun 17, 2021
8660fa5
add n5fsstore and tests
d-v-b Jul 5, 2021
3b341ed
resolve merge conflicts with main
d-v-b Jul 5, 2021
bb1121c
slightly smarter kwarg interception
d-v-b Jul 6, 2021
be8f37f
remove outdated unittest ref and fix the name of a test func
d-v-b Jul 6, 2021
95b2573
fix massive string block and fix default key_separator kwarg for FSStore
d-v-b Jul 6, 2021
ceba78d
flake8
d-v-b Jul 6, 2021
02ea91c
promote n5store to toplevel import and fix examples in docstring
d-v-b Jul 6, 2021
cb62c10
Merge branch 'master' into fix-dstore
joshmoore Aug 17, 2021
68adca5
Try fsspec 2021.7 (see #802)
joshmoore Aug 17, 2021
f2f75b7
Revert "Try fsspec 2021.7 (see #802)"
joshmoore Aug 17, 2021
930a821
Merge branch 'master' into n5fsstore
d-v-b Aug 17, 2021
a57b3bc
Add missing core tests for N5FSStore, and rchanges required for makin…
d-v-b Aug 17, 2021
9bb058f
Merge branch 'fix-dstore' of https://github.com/joshmoore/zarr-python…
d-v-b Aug 17, 2021
ee9cdbc
tmp: debug
joshmoore Aug 18, 2021
a853a29
uncomment N5 chunk ordering test
d-v-b Aug 18, 2021
7d3c879
more commented tests get uncommented
d-v-b Aug 18, 2021
f3ecd79
add dimension_separator to array metadata adaptor
d-v-b Aug 18, 2021
2d3d286
Merge branch 'fix-dstore' into pr-793+773
joshmoore Aug 19, 2021
5a105eb
Revert "tmp: debug"
joshmoore Aug 19, 2021
51b3109
Attempt failed: keeping '.' and switching
joshmoore Aug 19, 2021
aa75c98
Revert "Attempt failed: keeping '.' and switching"
joshmoore Aug 19, 2021
3daea7c
regex: attempt failed due to slight diff in files
joshmoore Aug 19, 2021
ce8a79e
Revert "regex: attempt failed due to slight diff in files"
joshmoore Aug 19, 2021
985c2a4
N5: use "." internally for dimension separation
joshmoore Aug 19, 2021
51836df
move FSSpec import guard
d-v-b Aug 19, 2021
3c5da2f
remove os.path.sep concatenation in listdir that was erroring a test,…
d-v-b Aug 19, 2021
eea4aaa
resolve merge conflicts in favor of upstream
d-v-b Sep 13, 2021
b8fe803
resolve merge conflicts in favor of upstream
d-v-b Sep 13, 2021
8fec1d6
make listdir implementation for n5fsstore look more like fsstore's li…
d-v-b Sep 14, 2021
46ebb44
Update hexdigest tests for N5Stores to account for the presence of th…
d-v-b Sep 15, 2021
864773d
Add tests for dimension_separator in array meta for N5Stores
d-v-b Sep 15, 2021
3b56155
N5FSStore: try to increase code coverage
joshmoore Sep 17, 2021
b0f6d33
flake8
d-v-b Sep 17, 2021
82ce89f
add chunk nesting test to N5FSStore test suite
d-v-b Sep 17, 2021
267c744
Merge branch 'master' into n5fsstore
joshmoore Sep 17, 2021
2b85410
make array_meta_key, group_meta_key, attrs_key private
d-v-b Sep 17, 2021
8bd6c41
Merge branch 'n5fsstore' of https://github.com/d-v-b/zarr-python into…
d-v-b Sep 17, 2021
aa4a723
N5FSStore: Remove ImportError test
joshmoore Sep 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions fixture/flat/.zarray
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"chunks": [
2,
2
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<i8",
"fill_value": 0,
"filters": null,
"order": "C",
"shape": [
2,
2
],
"zarr_format": 2
}
Binary file added fixture/flat/0.0
View file
Open in desktop
Binary file not shown.
23 changes: 23 additions & 0 deletions fixture/nested/.zarray
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"chunks": [
2,
2
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dimension_separator": "/",
"dtype": "<i8",
"fill_value": 0,
"filters": null,
"order": "C",
"shape": [
2,
2
],
"zarr_format": 2
}
Binary file added fixture/nested/0/0
View file
Open in desktop
Binary file not shown.
3 changes: 1 addition & 2 deletions requirements_dev_optional.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,5 @@ flake8==3.9.2
pytest-cov==2.12.1
pytest-doctestplus==0.10.1
h5py==3.3.0
s3fs==2021.6.0
fsspec==2021.6.0
fsspec==2021.7.0
moto[server]>=1.3.14
2 changes: 1 addition & 1 deletion zarr/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
zeros_like)
from zarr.errors import CopyError, MetadataError
from zarr.hierarchy import Group, group, open_group
from zarr.n5 import N5Store
from zarr.n5 import N5Store, N5FSStore
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This concerned me that it would need protecting by try/except block. In testing it, I realized FSStore only throws on __init__ and therefore N5FSStore could be less conservative. I've pushed:

aa4a723

from zarr.storage import (ABSStore, DBMStore, DictStore, DirectoryStore,
LMDBStore, LRUStoreCache, MemoryStore, MongoDBStore,
NestedDirectoryStore, RedisStore, SQLiteStore,
Expand Down
2 changes: 1 addition & 1 deletion zarr/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1952,7 +1952,7 @@ def _process_for_setitem(self, ckey, chunk_selection, value, fields=None):
return self._encode_chunk(chunk)

def _chunk_key(self, chunk_coords):
return self._key_prefix + '.'.join(map(str, chunk_coords))
return self._key_prefix + self._dimension_separator.join(map(str, chunk_coords))

def _decode_chunk(self, cdata, start=None, nitems=None, expected_shape=None):
# decompress
Expand Down
Loading