fixed length subtype recarray + "auto" shards crashes

[ilan-gold]

Zarr version

3.1.4.dev29+gfc8e8ad1a

Numcodecs version

0.16.3

Python Version

3.12.3

Operating System

macOS-15.1-arm64-arm-64bit

Installation

uv pip

Description

The reproducer below fails with shards="auto" but works otherwise.

Here is the traceback:

/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/array.py:4674: ZarrUserWarning: Automatic shard shape inference is experimental and may change without notice.
  shard_shape_parsed, chunk_shape_parsed = _auto_partition(
/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/dtype/npy/structured.py:318: UnstableSpecificationWarning: The data type (Structured(fields=(('PyvCr', FixedLengthUTF32(length=4, endianness='little')), ('UWJNo', FixedLengthUTF32(length=4, endianness='little'))))) does not have a Zarr V3 specification. That means that the representation of arrays saved with this data type may change without warning in a future version of Zarr Python. Arrays stored with this data type may be unreadable by other Zarr libraries. Use this data type at your own risk! Check https://github.com/zarr-developers/zarr-extensions/tree/main/data-types for the status of data type specifications for Zarr V3.
  v3_unstable_dtype_warning(self)
/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/dtype/npy/string.py:249: UnstableSpecificationWarning: The data type (FixedLengthUTF32(length=4, endianness='little')) does not have a Zarr V3 specification. That means that the representation of arrays saved with this data type may change without warning in a future version of Zarr Python. Arrays stored with this data type may be unreadable by other Zarr libraries. Use this data type at your own risk! Check https://github.com/zarr-developers/zarr-extensions/tree/main/data-types for the status of data type specifications for Zarr V3.
  v3_unstable_dtype_warning(self)
Traceback (most recent call last):
  File "/Users/ilangold/Projects/Theis/anndata/new_tester.py", line 61, in <module>
    f[...] = arr
    ~^^^^^
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/array.py", line 2966, in __setitem__
    self.set_basic_selection(cast("BasicSelection", pure_selection), value, fields=fields)
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/array.py", line 3200, in set_basic_selection
    sync(self._async_array._set_selection(indexer, value, fields=fields, prototype=prototype))
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/sync.py", line 159, in sync
    raise return_result
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/sync.py", line 119, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/array.py", line 1735, in _set_selection
    await self.codec_pipeline.write(
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/codec_pipeline.py", line 486, in write
    await concurrent_map(
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/common.py", line 100, in concurrent_map
    return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/common.py", line 98, in run
    return await func(*item)
           ^^^^^^^^^^^^^^^^^
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/codec_pipeline.py", line 352, in write_batch
    await self.encode_partial_batch(
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/codec_pipeline.py", line 247, in encode_partial_batch
    await self.array_bytes_codec.encode_partial(batch_info)
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/abc/codec.py", line 265, in encode_partial
    await concurrent_map(
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/common.py", line 100, in concurrent_map
    return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/core/common.py", line 98, in run
    return await func(*item)
           ^^^^^^^^^^^^^^^^^
  File "/Users/ilangold/Library/Caches/uv/environments-v2/new-tester-8e6728b281f68c98/lib/python3.12/site-packages/zarr/codecs/sharding.py", line 603, in _encode_partial_single
    chunks_per_shard = self._get_chunks_per_shard(shard_spec)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 3, in __hash__
TypeError: unhashable type: 'writeable void-scalar'

Steps to reproduce

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
#   "numpy",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues
from __future__ import annotations

import numpy as np
import zarr

# your reproducer code
zarr.print_debug_info()

arr = np.rec.array([('sQF', 'SQC'), ('XVut', 'XNsc'), ('HBz', 'xRL'),
           ('fuf', 'pyld'), ('Osuh', 'tRF'), ('PIpC', 'zzN'),
           ('YDyZ', 'MlJ'), ('RnG', 'PdF'), ('AHQ', 'uSc'),
           ('sRh', 'spmy')],
          dtype=[('btHIM', '<U4'), ('HLuXc', '<U4')])

g = zarr.open("foo.zarr", mode="w")
f = g.create_array("rec", shape=arr.shape, dtype=arr.dtype, shards="auto")
f[...] = arr

Additional output

No response

[d-v-b]

my first guess, based on the traceback, is that a numpy dtype is getting used in place of a zarr data type.

[ilan-gold]

Is it necessary to use a zarr data type here? From https://zarr.readthedocs.io/en/stable/user-guide/data_types.html#data-types-in-zarr-python

In this context, a “native” data type is a Python class, typically defined in another library, that models an array’s data type. For example, np.dtypes.UInt8DType is a native data type defined in NumPy. Zarr Python wraps the NumPy uint8 with a ZDType instance called [UInt8](https://zarr.readthedocs.io/en/stable/api/zarr/dtype/index.html#zarr.dtype.ZDType).

As of this writing, the only native data types Zarr Python supports are NumPy data types.

would make me think it would be supported natively as a python class, as passed in here. And this works without sharding,, it's just shards="auto".

I also tried dtype=[("btHIM", "<U4"), ("HLuXc", "<U4")] as in this section and dtye=zarr.dtype.parse_dtype(arr.dtype, zarr_format=3) to no avail

[d-v-b]

this looks relevant:

zarr-python/src/zarr/codecs/sharding.py

Lines 362 to 367 in fe42655

	# numpy void scalars are not hashable, which means an array spec with a fill value that is
	# a numpy void scalar will break the lru_cache. This is commented for now but should be
	# fixed. See https://github.com/zarr-developers/zarr-python/issues/3054
	# object.__setattr__(self, "_get_chunk_spec", lru_cache()(self._get_chunk_spec))
	object.__setattr__(self, "_get_index_chunk_spec", lru_cache()(self._get_index_chunk_spec))
	object.__setattr__(self, "_get_chunks_per_shard", lru_cache()(self._get_chunks_per_shard))

[ilan-gold]

Nice ok! I guess this issue is a bit of a no-op then for now it looks like. Feel free to close if it's duplicating the other then.

[d-v-b]

i do think it's worth having this issue as indicating either a new cached attribute we need to comment out 😒 or even more reason to fix the underlying issue