Skip to content

Reworked codec pipelines #1670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
May 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
9405fda
merge
normanrz Feb 20, 2024
019ecc8
refactors CodecPipelines
normanrz Apr 19, 2024
bd2160d
fixes
normanrz Apr 30, 2024
4887c29
adds HybridCodecPipeline
normanrz Apr 30, 2024
c3e3504
fixes
normanrz Apr 30, 2024
a578d95
typing
normanrz May 8, 2024
13212b5
merge
normanrz May 8, 2024
e3cad7c
typing
normanrz May 8, 2024
027ebb5
consistent naming
normanrz May 8, 2024
2bb00ae
Apply suggestions from code review
normanrz May 10, 2024
56877ee
encode/decode are batched by default
normanrz May 10, 2024
797b50b
merge
normanrz May 13, 2024
99a1f93
use zarr.config for batch_size
normanrz May 13, 2024
6d00972
merge
normanrz May 14, 2024
530e88b
don't use global lru_cache
normanrz May 14, 2024
9eda592
removes HybridCodecPipeline
normanrz May 15, 2024
d9aa24f
generic codec classes
normanrz May 15, 2024
a5fb71e
default batch size = 1
normanrz May 15, 2024
efd9bce
default batch size = 1
normanrz May 15, 2024
38c436d
docs
normanrz May 15, 2024
2397d3f
merge
normanrz May 15, 2024
1ad9896
Merge remote-tracking branch 'origin/v3' into batched-codec-pipeline
normanrz May 15, 2024
3a85a0a
Update src/zarr/codecs/batched_codec_pipeline.py
normanrz May 16, 2024
f33e66a
mv batched_codec_pipeline -> pipeline
normanrz May 16, 2024
9e42076
Merge branch 'batched-codec-pipeline' of github.com:zarr-developers/z…
normanrz May 16, 2024
95ae4b6
Remove ArrayV2 (#1857)
normanrz May 16, 2024
faa965d
Merge branch 'batched-codec-pipeline' of github.com:zarr-developers/z…
normanrz May 16, 2024
9bb3243
merge
normanrz May 16, 2024
db97439
merge
normanrz May 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -207,10 +207,6 @@ check_untyped_defs = false
module = [
"zarr.v2.*",
"zarr.abc.codec",
"zarr.codecs.bytes",
"zarr.codecs.pipeline",
"zarr.codecs.sharding",
"zarr.codecs.transpose",
"zarr.array_v2",
]
disallow_any_generics = false
Expand Down
11 changes: 2 additions & 9 deletions src/zarr/__init__.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
from __future__ import annotations

from typing import Union

import zarr.codecs # noqa: F401
from zarr.array import Array, AsyncArray
from zarr.array_v2 import ArrayV2
from zarr.config import config # noqa: F401
from zarr.group import AsyncGroup, Group
from zarr.store import (
Expand All @@ -18,19 +15,15 @@
assert not __version__.startswith("0.0.0")


async def open_auto_async(
store: StoreLike,
) -> Union[AsyncArray, AsyncGroup]:
async def open_auto_async(store: StoreLike) -> AsyncArray | AsyncGroup:
store_path = make_store_path(store)
try:
return await AsyncArray.open(store_path)
except KeyError:
return await AsyncGroup.open(store_path)


def open_auto(
store: StoreLike,
) -> Union[Array, ArrayV2, Group]:
def open_auto(store: StoreLike) -> Array | Group:
object = _sync(
open_auto_async(store),
)
Expand Down
Loading
Loading