Skip to content

Raise an error when doing rolling window operations with dask #3036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/whats-new.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ Enhancements
Bug fixes
~~~~~~~~~

- Rolling operations on xarray objects containing dask arrays could silently
compute the incorrect result or use large amounts of memory (:issue:`2940`).
For now, these operations have been disabled with an explicit error.
By `Stephan Hoyer <https://github.com/shoyer>`_.
- NetCDF4 output: variables with unlimited dimensions must be chunked (not
contiguous) on output. (:issue:`1849`)
By `James McCreight <https://github.com/jmccreight>`_.
Expand Down
7 changes: 7 additions & 0 deletions xarray/core/dask_array_ops.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,13 @@

def dask_rolling_wrapper(moving_func, a, window, min_count=None, axis=-1):
'''wrapper to apply bottleneck moving window funcs on dask arrays'''

raise NotImplementedError(
'rolling operations on xarray objects backed by dask arrays have not '
'been implemented yet (https://github.com/pydata/xarray/issues/2940). '
'For now, load your arrays into memory for rolling window operations '
'by calling .compute().')

dtype, fill_value = dtypes.maybe_promote(a.dtype)
a = a.astype(dtype)
# inputs for overlap
Expand Down
13 changes: 10 additions & 3 deletions xarray/tests/test_dataarray.py
Original file line number Diff line number Diff line change
Expand Up @@ -3724,9 +3724,16 @@ def test_rolling_wrapped_dask_nochunk(center):

da_day_clim = xr.DataArray(np.arange(1, 367),
coords=[np.arange(1, 367)], dims='dayofyear')
expected = da_day_clim.rolling(dayofyear=31, center=center).mean()
actual = da_day_clim.chunk().rolling(dayofyear=31, center=center).mean()
assert_allclose(actual, expected)
expected = da_day_clim.rolling(dayofyear=31, center=center).mean() # noqa
actual = da_day_clim.chunk().rolling( # noqa
dayofyear=31, center=center).mean()
print(actual)
with pytest.raises(NotImplementedError):
actual = da_day_clim.chunk().rolling( # noqa
dayofyear=31, center=center).mean()
# TODO: uncomment this assertion once we fix rolling window operations with
# dask (https://github.com/pydata/xarray/issues/2940)
# assert_allclose(actual, expected)


@pytest.mark.parametrize('center', (True, False))
Expand Down