Merge branch 'main' into FSStore_ensure_bytes

madsbk · web-flow · commit 3afda2b7ae18 · 2023-01-16T17:02:30.000+01:00
diff --git a/.github/workflows/needs_release_notes.yml b/.github/workflows/needs_release_notes.yml
@@ -5,6 +5,7 @@ on:
 
 jobs:
   triage:
+    if: ${{ github.event.pull_request.user.login != 'dependabot[bot]' }} && ${{ github.event.pull_request.user.login != 'pre-commit-ci[bot]' }}
     runs-on: ubuntu-latest
     steps:
     - uses: actions/labeler@main
diff --git a/docs/release.rst b/docs/release.rst
@@ -6,19 +6,67 @@ Release notes
     # to document your changes. On releases it will be
     # re-indented so that it does not show up in the notes.
 
-.. _unreleased:
+    .. _unreleased:
 
-Unreleased
-----------
+    Unreleased
+    ----------
 ..
     # .. warning::
     #    Pre-release! Use :command:`pip install --pre zarr` to evaluate this release.
 
+.. _release_2.13.4:
+
+2.13.5
+------
+
+Bug fixes
+~~~~~~~~~
+
+* Ensure ``zarr.create`` uses writeable mode to fix issue with :issue:`1304`.
+  By :user:`James Bourbeau <jrbourbeau>` :issue:`1309`.
+
+2.13.4
+------
+
+Appreciation
+~~~~~~~~~~~~~
+
+Special thanks to Outreachy participants for contributing to most of the maintenance PRs. Please read the blog post summarising the contribution phase and welcoming new Outreachy interns: https://zarr.dev/blog/welcoming-outreachy-2022-interns/
+
+
+Enhancements
+~~~~~~~~~~~~
+
+* Handle fsspec.FSMap using FSStore store.
+  By :user:`Rafal Wojdyla <ravwojdyla>` :issue:`1304`.
+
+Bug fixes
+~~~~~~~~~
+
+* Fix bug that caused double counting of groups in ``groups()`` and ``group_keys()`` methods with V3 stores.
+  By :user:`Ryan Abernathey <rabernat>` :issue:`1228`.
+
+* Remove unnecessary calling of `contains_array` for key that ended in `.array.json`.
+  By :user:`Joe Hamman <jhamman>` :issue:`1149`.
+
 * Fix bug that caused double counting of groups in ``groups()`` and ``group_keys()``
   methods with V3 stores.
   By :user:`Ryan Abernathey <rabernat>` :issue:`1228`.
 
-.. _release_2.13.2:
+Documentation
+~~~~~~~~~~~~~
+
+* Fix minor indexing errors in tutorial and specification examples of documentation. 
+  By :user:`Kola Babalola <sprynt001>` :issue:`1277`.
+
+* Add `requirements_rtfd.txt` in `contributing.rst`.
+  By :user:`AWA BRANDON AWA <DON-BRAN>` :issue:`1243`.
+
+* Add documentation for find/findall using visit.
+  By :user:`Weddy Gikunda <caviere>` :issue:`1241`.
+
+* Refresh of the main landing page.
+  By :user:`Josh Moore <joshmoore>` :issue:`1173`.
 
 Maintenance
 ~~~~~~~~~~~
@@ -53,6 +101,77 @@ Maintenance
 * Delete unused files.
   By :user:`John Kirkham <jakirkham>` :issue:`1251`.
 
+* Skip labeller for bot PRs.
+  By :user:`Saransh Chopra <Saransh-cpp>` :issue:`1271`.
+
+* Restore Flake8 configuration.
+  By :user:`John Kirkham <jakirkham>` :issue:`1249`.
+
+* Add missing newline at EOF.
+  By :user:`Dimitri Papadopoulos` :issue:`1253`.
+
+* Add `license_files` to `pyproject.toml`.
+  By :user:`John Kirkham <jakirkham>` :issue:`1247`.
+
+* Adding `pyupgrade` suggestions.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1225`.
+
+* Fixed some linting errors.
+  By :user:`Weddy Gikunda <caviere>` :issue:`1226`.
+
+* Added the link to main website in readthedocs sidebar.
+  By :user:`Stephanie_nkwatoh <steph237>` :issue:`1216`.
+
+* Remove redundant wheel dependency in `pyproject.toml`.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1233`.
+
+* Turned on `isloated_build` in `tox.ini` file.
+  By :user:`AWA BRANDON AWA <DON-BRAN>` :issue:`1210`.
+
+* Fixed `flake8` alert and avoid duplication of `Zarr Developers`.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1203`.
+
+* Bump to NumPy 1.20+ in `environment.yml`.
+  By :user:`John Kirkham <jakirkham>` :issue:`1201`.
+
+* Bump to NumPy 1.20 in `pyproject.toml`.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1192`.
+
+* Remove LGTM (`.lgtm.yml`) configuration file.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1191`.
+
+* Codespell will skip `fixture` in pre-commit.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1197`.
+
+* Add msgpack in `requirements_rtfd.txt`.
+  By :user:`Emmanuel Bolarinwa <GbotemiB>` :issue:`1188`.
+
+* Added license to docs fixed a typo from `_spec_v2` to `_spec_v3`.
+  By :user:`AWA BRANDON AWA <DON-BRAN>` :issue:`1182`.
+
+* Fixed installation link in `README.md`.
+  By :user:`AWA BRANDON AWA <DON-BRAN>` :issue:`1177`.
+
+* Fixed typos in `installation.rst` and `release.rst`.
+  By :user:`Chizoba Nweke <zobbs-git>` :issue:`1178`.
+
+* Set `docs/conf.py` language to `en`.
+  By :user:`AWA BRANDON AWA <DON-BRAN>` :issue:`1174`.
+
+* Added `installation.rst` to the docs.
+  By :user:`AWA BRANDON AWA <DON-BRAN>` :issue:`1170`.
+
+* Adjustment of year to `2015-2018` to `2015-2022` in the docs.
+  By :user:`Emmanuel Bolarinwa <GbotemiB>` :issue:`1165`.
+
+* Updated `Forking the repository` section in `contributing.rst`.
+  By :user:`AWA BRANDON AWA <DON-BRAN>` :issue:`1171`.
+
+* Updated GitHub actions.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1134`.
+
+* Uopdate web links: `http:// → https://`.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1313`.
 
 .. _release_2.13.3:
 
diff --git a/docs/spec/v1.rst b/docs/spec/v1.rst
@@ -144,9 +144,9 @@ are converted to strings and concatenated with the period character
 ('.') separating each index. For example, given an array with shape
 (10000, 10000) and chunk shape (1000, 1000) there will be 100 chunks
 laid out in a 10 by 10 grid. The chunk with indices (0, 0) provides
-data for rows 0-1000 and columns 0-1000 and is stored under the key
-'0.0'; the chunk with indices (2, 4) provides data for rows 2000-3000
-and columns 4000-5000 and is stored under the key '2.4'; etc.
+data for rows 0-999 and columns 0-999 and is stored under the key
+'0.0'; the chunk with indices (2, 4) provides data for rows 2000-2999
+and columns 4000-4999 and is stored under the key '2.4'; etc.
 
 There is no need for all chunks to be present within an array
 store. If a chunk is not present then it is considered to be in an
diff --git a/docs/spec/v2.rst b/docs/spec/v2.rst
@@ -216,9 +216,9 @@ To form a string key for a chunk, the indices are converted to strings and
 concatenated with the period character (".") separating each index. For
 example, given an array with shape (10000, 10000) and chunk shape (1000, 1000)
 there will be 100 chunks laid out in a 10 by 10 grid. The chunk with indices
-(0, 0) provides data for rows 0-1000 and columns 0-1000 and is stored under the
-key "0.0"; the chunk with indices (2, 4) provides data for rows 2000-3000 and
-columns 4000-5000 and is stored under the key "2.4"; etc.
+(0, 0) provides data for rows 0-999 and columns 0-999 and is stored under the
+key "0.0"; the chunk with indices (2, 4) provides data for rows 2000-2999 and
+columns 4000-4999 and is stored under the key "2.4"; etc.
 
 There is no need for all chunks to be present within an array store. If a chunk
 is not present then it is considered to be in an uninitialized state.  An
diff --git a/docs/tutorial.rst b/docs/tutorial.rst
@@ -525,9 +525,9 @@ When the indexing arrays have different shapes, they are broadcast together.
 That is, the following two calls are equivalent::
 
     >>> z[1, [1, 3]]
-    array([5, 7])
+    array([6, 8])
     >>> z[[1, 1], [1, 3]]
-    array([5, 7])
+    array([6, 8])
 
 Indexing with a mask array
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/notebooks/dask_copy.ipynb b/notebooks/dask_copy.ipynb
@@ -33,7 +33,7 @@
       "text/html": [
        "\n",
        "    <div class=\"bk-root\">\n",
-       "        <a href=\"http://bokeh.pydata.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",
+       "        <a href=\"https://bokeh.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",
        "        <span id=\"d4821cb3-378c-411d-a941-d0708c0c532b\">Loading BokehJS ...</span>\n",
        "    </div>"
       ]
diff --git a/notebooks/dask_count_alleles.ipynb b/notebooks/dask_count_alleles.ipynb
@@ -26,7 +26,7 @@
       "text/html": [
        "\n",
        "    <div class=\"bk-root\">\n",
-       "        <a href=\"http://bokeh.pydata.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",
+       "        <a href=\"https://bokeh.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",
        "        <span id=\"b153ad5f-436a-4afb-945c-87790add89c8\">Loading BokehJS ...</span>\n",
        "    </div>"
       ]
diff --git a/requirements_dev_minimal.txt b/requirements_dev_minimal.txt
@@ -3,6 +3,6 @@ asciitree==0.3.3
 fasteners==0.18
 numcodecs==0.11.0
 msgpack-python==0.5.6
-setuptools-scm==7.0.5
+setuptools-scm==7.1.0
 # test requirements
-pytest==7.2.0
+pytest==7.2.1
diff --git a/requirements_dev_numpy.txt b/requirements_dev_numpy.txt
@@ -1,4 +1,4 @@
 # Break this out into a separate file to allow testing against
 # different versions of numpy. This file should pin to the latest
 # numpy version.
-numpy==1.23.5
+numpy==1.24.1
diff --git a/requirements_dev_optional.txt b/requirements_dev_optional.txt
@@ -1,20 +1,19 @@
 # optional library requirements
 # bsddb3==6.2.6; sys_platform != 'win32'
-lmdb==1.3.0; sys_platform != 'win32'
+lmdb==1.4.0; sys_platform != 'win32'
 # optional library requirements for Jupyter
 ipytree==0.2.2
-ipywidgets==8.0.2
+ipywidgets==8.0.4
 # optional library requirements for services
 # don't let pyup change pinning for azure-storage-blob, need to pin to older
 # version to get compatibility with azure storage emulator on appveyor (FIXME)
 azure-storage-blob==12.14.1 # pyup: ignore
-redis==4.4.0
+redis==4.4.2
 types-redis
 types-setuptools
 pymongo==4.3.3
 # optional test requirements
 coverage
-flake8==5.0.4
 pytest-cov==4.0.0
 pytest-doctestplus==0.12.1
 pytest-timeout==2.1.0
diff --git a/zarr/_storage/v3.py b/zarr/_storage/v3.py
@@ -567,6 +567,16 @@ def _normalize_store_arg_v3(store: Any, storage_options=None, mode="r") -> BaseS
         return store
     if isinstance(store, os.PathLike):
         store = os.fspath(store)
+    if FSStore._fsspec_installed():
+        import fsspec
+        if isinstance(store, fsspec.FSMap):
+            return FSStoreV3(store.root,
+                             fs=store.fs,
+                             mode=mode,
+                             check=store.check,
+                             create=store.create,
+                             missing_exceptions=store.missing_exceptions,
+                             **(storage_options or {}))
     if isinstance(store, str):
         if "://" in store or "::" in store:
             store = FSStoreV3(store, mode=mode, **(storage_options or {}))
diff --git a/zarr/creation.py b/zarr/creation.py
@@ -145,7 +145,7 @@ def create(shape, chunks=True, dtype=None, compressor='default',
         zarr_version = getattr(chunk_store, '_store_version', DEFAULT_ZARR_VERSION)
 
     # handle polymorphic store arg
-    store = normalize_store_arg(store, zarr_version=zarr_version)
+    store = normalize_store_arg(store, zarr_version=zarr_version, mode="w")
     zarr_version = getattr(store, '_store_version', DEFAULT_ZARR_VERSION)
 
     # API compatibility with h5py
diff --git a/zarr/storage.py b/zarr/storage.py
@@ -139,6 +139,16 @@ def _normalize_store_arg_v2(store: Any, storage_options=None, mode="r") -> BaseS
         return store
     if isinstance(store, os.PathLike):
         store = os.fspath(store)
+    if FSStore._fsspec_installed():
+        import fsspec
+        if isinstance(store, fsspec.FSMap):
+            return FSStore(store.root,
+                           fs=store.fs,
+                           mode=mode,
+                           check=store.check,
+                           create=store.create,
+                           missing_exceptions=store.missing_exceptions,
+                           **(storage_options or {}))
     if isinstance(store, str):
         if "://" in store or "::" in store:
             return FSStore(store, mode=mode, **(storage_options or {}))
@@ -1308,6 +1318,8 @@ def __init__(self, url, normalize_keys=False, key_separator=None,
                  create=False,
                  missing_exceptions=None,
                  **storage_options):
+        if not self._fsspec_installed():  # pragma: no cover
+            raise ImportError("`fsspec` is required to use zarr's FSStore")
         import fsspec
 
         mapper_options = {"check": check, "create": create}
@@ -1485,6 +1497,13 @@ def clear(self):
             raise ReadOnlyError()
         self.map.clear()
 
+    @classmethod
+    def _fsspec_installed(cls):
+        """Returns true if fsspec is installed"""
+        import importlib.util
+
+        return importlib.util.find_spec("fsspec") is not None
+
 
 class TempStore(DirectoryStore):
     """Directory store using a temporary directory for storage.
diff --git a/zarr/tests/test_creation.py b/zarr/tests/test_creation.py
@@ -19,7 +19,7 @@
 from zarr._storage.store import v3_api_available
 from zarr._storage.v3 import DirectoryStoreV3, KVStoreV3
 from zarr.sync import ThreadSynchronizer
-from zarr.tests.util import mktemp
+from zarr.tests.util import mktemp, have_fsspec
 
 _VERSIONS = ((None, 2, 3) if v3_api_available else (None, 2))
 _VERSIONS2 = ((2, 3) if v3_api_available else (2, ))
@@ -429,6 +429,18 @@ def test_create_in_dict(zarr_version, at_root):
     assert isinstance(a.store, expected_store_type)
 
 
+@pytest.mark.skipif(have_fsspec is False, reason="needs fsspec")
+@pytest.mark.parametrize('zarr_version', _VERSIONS)
+@pytest.mark.parametrize('at_root', [False, True])
+def test_create_writeable_mode(zarr_version, at_root, tmp_path):
+    # Regression test for https://github.com/zarr-developers/zarr-python/issues/1306
+    import fsspec
+    kwargs = _init_creation_kwargs(zarr_version, at_root)
+    store = fsspec.get_mapper(str(tmp_path))
+    z = create(100, store=store, **kwargs)
+    assert z.store.map == store
+
+
 @pytest.mark.parametrize('zarr_version', _VERSIONS)
 @pytest.mark.parametrize('at_root', [False, True])
 def test_empty_like(zarr_version, at_root):
diff --git a/zarr/tests/test_storage.py b/zarr/tests/test_storage.py
@@ -2556,10 +2556,15 @@ def test_normalize_store_arg(tmpdir):
         assert isinstance(store, Class)
 
     if have_fsspec:
+        import fsspec
+
         path = tempfile.mkdtemp()
         store = normalize_store_arg("file://" + path, zarr_version=2, mode='w')
         assert isinstance(store, FSStore)
 
+        store = normalize_store_arg(fsspec.get_mapper("file://" + path))
+        assert isinstance(store, FSStore)
+
 
 def test_meta_prefix_6853():
 
diff --git a/zarr/tests/test_storage_v3.py b/zarr/tests/test_storage_v3.py
@@ -467,11 +467,16 @@ def test_normalize_store_arg_v3(tmpdir):
         normalize_store_arg(str(fn), zarr_version=3, mode='w', storage_options={"some": "kwargs"})
 
     if have_fsspec:
+        import fsspec
+
         path = tempfile.mkdtemp()
         store = normalize_store_arg("file://" + path, zarr_version=3, mode='w')
         assert isinstance(store, FSStoreV3)
         assert 'zarr.json' in store
 
+        store = normalize_store_arg(fsspec.get_mapper("file://" + path), zarr_version=3)
+        assert isinstance(store, FSStoreV3)
+
     fn = tmpdir.join('store.n5')
     with pytest.raises(NotImplementedError):
         normalize_store_arg(str(fn), zarr_version=3, mode='w')

Original file line number	Diff line number	Diff line change
`@@ -33,7 +33,7 @@`
`33`	`33`	`"text/html": [`
`34`	`34`	`"\n",`
`35`	`35`	`" <div class=\"bk-root\">\n",`
`36`		`- " <a href=\"http://bokeh.pydata.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",`
	`36`	`+ " <a href=\"https://bokeh.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",`
`37`	`37`	`" <span id=\"d4821cb3-378c-411d-a941-d0708c0c532b\">Loading BokehJS ...</span>\n",`
`38`	`38`	`" </div>"`
`39`	`39`	`]`
Original file line number	Diff line number	Diff line change
`@@ -26,7 +26,7 @@`
`26`	`26`	`"text/html": [`
`27`	`27`	`"\n",`
`28`	`28`	`" <div class=\"bk-root\">\n",`
`29`		`- " <a href=\"http://bokeh.pydata.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",`
	`29`	`+ " <a href=\"https://bokeh.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",`
`30`	`30`	`" <span id=\"b153ad5f-436a-4afb-945c-87790add89c8\">Loading BokehJS ...</span>\n",`
`31`	`31`	`" </div>"`
`32`	`32`	`]`