Test failure on i386: ValueError: bins must be monotonically increasing or decreasing

[avalentino]

I'm trying to update the Debian package for flox to the latest version (flox 0.7.2).
On i386 platforms I get the following error that was not present in flox v0.6.7:

python3.11 -m pytest -k "not test_xarray_resample and not test_xarray_reduce_single_grouper and not test_func_is_aggregation and not test_groupby_bins_indexed_coordinate and not test_multi_index_groupby_sum" /home/antonio/debian/git/build-area/flox-0.7.2/tests
============================= test session starts ==============================
platform linux -- Python 3.11.4, pytest-7.2.1, pluggy-1.0.0+repack
rootdir: /home/antonio/debian/git/build-area/flox-0.7.2, configfile: pyproject.toml
collected 8724 items / 10 deselected / 8714 selected

../../../tests/test_core.py ...ss..........ss..........ss..........ss... [  0%]

[CUT]

=================================== FAILURES ===================================
__________________ test_group_by_datetime[flox-split-reduce] ___________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
___________________ test_group_by_datetime[flox-map-reduce] ____________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
_____________________ test_group_by_datetime[flox-cohorts] _____________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
__________________ test_group_by_datetime[numpy-split-reduce] __________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
___________________ test_group_by_datetime[numpy-map-reduce] ___________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
____________________ test_group_by_datetime[numpy-cohorts] _____________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
__________________ test_group_by_datetime[numba-split-reduce] __________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
___________________ test_group_by_datetime[numba-map-reduce] ___________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
____________________ test_group_by_datetime[numba-cohorts] _____________________
../../../tests/test_core.py:1021: in test_group_by_datetime
    actual, _ = groupby_reduce(daskarray, t.to_numpy(), isbin=True, expected_groups=edges, **kwargs)
../../../flox/core.py:1869: in groupby_reduce
    bys, final_groups, grp_shape = _factorize_multiple(
../../../flox/core.py:1659: in _factorize_multiple
    group_idx, found_groups, grp_shape, ngroups, size, props = factorize_(
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
____________________________ test_datetime_binning _____________________________
../../../tests/test_core.py:967: in test_datetime_binning
    ret = factorize_((by.to_numpy(),), axes=(0,), expected_groups=(actual,))
../../../flox/core.py:538: in factorize_
    idx = np.digitize(
<__array_function__ internals>:200: in digitize
    ???
/usr/lib/python3/dist-packages/numpy/lib/function_base.py:5606: in digitize
    raise ValueError("bins must be monotonically increasing or decreasing")
E   ValueError: bins must be monotonically increasing or decreasing
=============================== warnings summary ===============================
../../../../../../../../../usr/lib/python3/dist-packages/llvmlite/binding/ffi.py:159
  /usr/lib/python3/dist-packages/llvmlite/binding/ffi.py:159: DeprecationWarning: path is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice.
    _lib_handle = importlib.resources.path(pkgname, _lib_name)

tests/test_xarray.py::test_preserve_multiindex
  /usr/lib/python3/dist-packages/xarray/core/dataarray.py:3017: DeprecationWarning: Deleting a single level of a MultiIndex is deprecated. Previously, this deleted all levels of a MultiIndex. Please also drop the following variables: {'region'} to avoid an error in the future.
    ds = self._to_temp_dataset().drop_vars(names, errors=errors)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED ../../../tests/test_core.py::test_group_by_datetime[flox-split-reduce]
FAILED ../../../tests/test_core.py::test_group_by_datetime[flox-map-reduce]
FAILED ../../../tests/test_core.py::test_group_by_datetime[flox-cohorts] - Va...
FAILED ../../../tests/test_core.py::test_group_by_datetime[numpy-split-reduce]
FAILED ../../../tests/test_core.py::test_group_by_datetime[numpy-map-reduce]
FAILED ../../../tests/test_core.py::test_group_by_datetime[numpy-cohorts] - V...
FAILED ../../../tests/test_core.py::test_group_by_datetime[numba-split-reduce]
FAILED ../../../tests/test_core.py::test_group_by_datetime[numba-map-reduce]
FAILED ../../../tests/test_core.py::test_group_by_datetime[numba-cohorts] - V...
FAILED ../../../tests/test_core.py::test_datetime_binning - ValueError: bins ...
= 10 failed, 7022 passed, 1475 skipped, 10 deselected, 204 xfailed, 3 xpassed, 2 warnings in 939.47s (0:15:39) =

I have still not looked into the issue but I would appreciate any hint about how to solve this issue.

[dcherian]

Thanks. I don't know what that happens.

test_datetime_binning is definitely testing the core issue, so you can just execute that for quick iteration.

Here's what actual is on my machine

IntervalIndex([(2010-08-01, 2010-08-02], (2010-08-02, 2010-08-03], (2010-08-03, 2010-08-04], (2010-08-04, 2010-08-05], (2010-08-05, 2010-08-06] ... (2010-08-10, 2010-08-11], (2010-08-11, 2010-08-12], (2010-08-12, 2010-08-13], (2010-08-13, 2010-08-14], (2010-08-14, 2010-08-15]], dtype='interval[datetime64[ns], right]')

Is it overflowing here

flox/flox/core.py

Line 540 in 5d223a7

bins=bins.view(np.intp) if bins.dtype.kind == "M" else bins,

Maybe change intp to int64?

[avalentino]

OK, the value of actual is the same, but I can confirm that changing intp to int64 in the line indicated above fixes the problem.

Thanks a lot @dcherian

[dcherian]

PR welcome!

[avalentino]

Done