Structured Arrays

[jakirkham]

Was interested in the possibility of storing structured arrays (a.k.a. record arrays or compound arrays) using Zarr. This is sort of related to PR ( https://github.com/alimanfoo/zarr/pull/84 ), but structured arrays are a simpler type. It also corresponds to a NumPy array type and a HDF5 dataset. So it might make sense to add similar support in Zarr. OTOH in both HDF5 and Zarr it is possible to construct a group that contains the individual arrays and at least with HDF5 this makes it easier to view using HDFView. Am opening this issue to discuss and weigh different options regarding the storage of record arrays using Zarr.

ref: https://docs.scipy.org/doc/numpy/user/basics.rec.html
ref: https://support.hdfgroup.org/HDF5/Tutor/compound.html

[alimanfoo]

Zarr does support storing structured arrays, e.g.:

In [9]: import numpy as np

In [10]: import zarr

In [11]: a = np.array([(b'a', 1), (b'b', 2)], dtype=[('foo', 'S1'), ('bar', int)])

In [12]: z = zarr.array(a)

In [13]: z
Out[13]: 
Array((2,), [('foo', 'S1'), ('bar', '<i8')], chunks=(2,), order=C)
  nbytes: 18; nbytes_stored: 438; ratio: 0.0; initialized: 1/1
  compressor: Blosc(cname='lz4', clevel=5, shuffle=1)
  store: dict

However, one nice feature that h5py has which zarr doesn't currently have is the ability to load a specific field or fields, e.g., z[:, 'foo']. I have actually wanted this recently.

FWIW regarding the choice of single array with structured dtype versus one array with simple dtype per column, I've generally found the latter (i.e., columnar) storage to be more flexible and more efficient for a variety of uses. However I still do use structured arrays too and so would be interested to support both patterns.

[jakirkham]

Oh, that's good to know. Then maybe I'm running into a bug or an unsupported edge case. ( https://github.com/alimanfoo/zarr/issues/111 ) Will close this out.

However, one nice feature that h5py has which zarr doesn't currently have is the ability to load a specific field or fields, e.g., z[:, 'foo']. I have actually wanted this recently.

Indeed, that would be a very nice feature for Zarr. Used this with h5py some myself.

Edit: Opened issue ( https://github.com/alimanfoo/zarr/issues/112 ) on this this point.

...I've generally found the latter (i.e., columnar) storage to be more flexible and more efficient for a variety of uses. However I still do use structured arrays too and so would be interested to support both patterns.

By columnar, I'm assuming you mean having a single type for the array. In which case, I do agree with you. Still sometimes a structured array is just the right data structure for the problem.