OverflowError if using NaT as fill_value for mM dtypes

[thomascoquet]

Code sample

import zarr
import numpy as np

z = zarr.full(dtype='m8[s]', 
              fill_value=np.timedelta64('nat'), 
              shape=10)

raises

OverflowError: int too big to convert
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/tom/anaconda3/lib/python3.6/site-packages/zarr/meta.py", line 38, in decode_array_metadata
    fill_value = decode_fill_value(meta['fill_value'], dtype)
  File "/home/tom/anaconda3/lib/python3.6/site-packages/zarr/meta.py", line 159, in decode_fill_value
    return np.array(v, dtype=dtype)[()]
SystemError: <built-in function array> returned a result with an error set
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/tom/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2963, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-156-d1f331fc15fd>", line 6, in <module>
    shape=10)
  File "/home/tom/anaconda3/lib/python3.6/site-packages/zarr/creation.py", line 274, in full
    return create(shape=shape, fill_value=fill_value, **kwargs)
  File "/home/tom/anaconda3/lib/python3.6/site-packages/zarr/creation.py", line 123, in create
    cache_metadata=cache_metadata, cache_attrs=cache_attrs, read_only=read_only)
  File "/home/tom/anaconda3/lib/python3.6/site-packages/zarr/core.py", line 123, in __init__
    self._load_metadata()
  File "/home/tom/anaconda3/lib/python3.6/site-packages/zarr/core.py", line 140, in _load_metadata
    self._load_metadata_nosync()
  File "/home/tom/anaconda3/lib/python3.6/site-packages/zarr/core.py", line 155, in _load_metadata_nosync
    meta = decode_array_metadata(meta_bytes)
  File "/home/tom/anaconda3/lib/python3.6/site-packages/zarr/meta.py", line 50, in decode_array_metadata
    raise MetadataError('error decoding metadata: %s' % e)
zarr.errors.MetadataError: error decoding metadata: <built-in function array> returned a result with an error set

Analysis

I think the serialization method for mM types does not allow to unserialize properly Not a Time fill values.

np.array(int(np.datetime64('nat').view('u8')), dtype='M8[D]')[()]

Resolution

In meta.py / encode_fill_value:

    elif dtype.kind in 'mM':
        if np.isnat(v):
            return 'NaT'
        else:
            return int(v.view('u8'))

[jakirkham]

What happens if you change u8 to f8?

[jakirkham]

The key point appears to be NaT is represented as -0. So casting to an unsigned integral type causes issues. If a signed integral type is used, this works without issues.

In [1]: import numpy as np                                                      

In [2]: np.array(int(np.datetime64('nat').view('i8')), dtype='M8[D]')[()]       
Out[2]: numpy.datetime64('NaT')

[jakirkham]

Have PR ( zarr-developers/numcodecs#127 ) and PR ( #344 ) in the works to fix this issue.

Edit: Have verified these fix the case described in the OP. The latter PR includes a round-trip test of NaT as a fill_value.

[thomascoquet]

Thank you for the fix!

[jakirkham]

Of course, were you able to give it a try?

[jakirkham]

Closing as this should be resolved with PR ( #344 ) and Numcodecs 0.6.2+. Please let us know if you run into any issues.