Skip to content

Xarray with cfgrib backend errors with .where() when drop=True #4733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
DanielAdriaansen opened this issue Dec 24, 2020 · 4 comments · Fixed by #4737
Closed

Xarray with cfgrib backend errors with .where() when drop=True #4733

DanielAdriaansen opened this issue Dec 24, 2020 · 4 comments · Fixed by #4737
Labels
topic-backends

Comments

@DanielAdriaansen
Copy link
Contributor

What happened:
When loading a HRRR GRIBv2 file in this manner:
ds = xr.open_dataset(gribfile_path,engine='cfgrib',backend_kwargs={'filter_by_keys':{'typeOfLevel':'hybrid'}})

I have trouble using the .where() method when drop=True. If I set drop=False, it works fine. I am attempting to subset via latitude and longitude like this:

ds_sub = ds.where((ds.latitude>=40.0)&(ds.latitude<=50.0)&(ds.longitude>=252.0)&(ds.longitude<=280.0),drop=True)

However I receive the following errors:

Traceback (most recent call last):
  File "read_interp_format_grib.py", line 77, in <module>
    test = ds.where(mask_lat & mask_lon,drop=True)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/common.py", line 1268, in where
    return ops.where_method(self, cond, other)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/ops.py", line 193, in where_method
    return apply_ufunc(
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/computation.py", line 1092, in apply_ufunc
    return apply_dataset_vfunc(
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/computation.py", line 410, in apply_dataset_vfunc
    result_vars = apply_dict_of_variables_vfunc(
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/computation.py", line 356, in apply_dict_of_variables_vfunc
    result_vars[name] = func(*variable_args)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/computation.py", line 606, in apply_variable_ufunc
    input_data = [
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/computation.py", line 607, in <listcomp>
    broadcast_compat_data(arg, broadcast_dims, core_dims)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/computation.py", line 522, in broadcast_compat_data
    data = variable.data
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/variable.py", line 359, in data
    return self.values
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/variable.py", line 510, in values
    return _as_array_or_item(self._data)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/variable.py", line 272, in _as_array_or_item
    data = np.asarray(data)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/indexing.py", line 685, in __array__
    self._ensure_cached()
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/indexing.py", line 682, in _ensure_cached
    self.array = NumpyIndexingAdapter(np.asarray(self.array))
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/indexing.py", line 655, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/indexing.py", line 560, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/backends/cfgrib_.py", line 23, in __getitem__
    return indexing.explicit_indexing_adapter(
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/indexing.py", line 848, in explicit_indexing_adapter
    result = NumpyIndexingAdapter(np.asarray(result))[numpy_indices]
  File "/d1/anaconda3/envs/era5/lib/python3.8/site-packages/xarray/core/indexing.py", line 1294, in __getitem__
    return array[key]
IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed

What you expected to happen:
I expect the dataset to be reduced in the x and y (latitude/longitude) dimensions where cond=False from .where() when drop=True.

Minimal Complete Verifiable Example:

# Put your MCVE code here

Anything else we need to know?:
I was able to confirm cfgrib is where the issue lies by doing the following:

ds = xr.open_dataset(gribfile_path,engine='cfgrib',backend_kwargs={'filter_by_keys':{'typeOfLevel':'hybrid'}})
ds.to_netcdf('test.nc')
dsnc = xr.open_dataset('test.nc')
ds_sub = dsnc.where((dsnc.latitude>=40.0)&(dsnc.latitude<=50.0)&(dsnc.longitude>=252.0)&(dsnc.longitude<=280.0),drop=True)

That correctly gives me:
Dimensions: (hybrid: 50, x: 792, y: 414)

Originally x=1799 and y=1059.

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.6 | packaged by conda-forge | (default, Oct 7 2020, 19:08:05)
[GCC 7.5.0]
python-bits: 64
OS: Linux
OS-release: 4.9.0-14-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.6
libnetcdf: 4.7.4

xarray: 0.16.1
pandas: 1.1.3
numpy: 1.19.1
scipy: 1.5.2
netCDF4: 1.5.5.1
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.2.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: 0.9.8.5
iris: None
bottleneck: None
dask: 2.30.0
distributed: 2.30.0
matplotlib: 3.3.2
cartopy: 0.17.0
seaborn: None
numbagg: None
pint: 0.16.1
setuptools: 49.6.0.post20200917
pip: 20.2.3
conda: None
pytest: None
IPython: None
sphinx: None
@DanielAdriaansen
Copy link
Contributor Author

Paging @alexamici?

@alexamici
Copy link
Collaborator

@DanielAdriaansen thanks for reporting, this is a duplicate of ecmwf/cfgrib#157 so I'm going to close this one, and you can follow there.

I'm a bit embarrassed as @aurghs identified this bug a while back, but I forgot about it!

The fix should be easy, I'll try to make a release shortly.

@alexamici
Copy link
Collaborator

Looks like an effective way to fix the bug is in xarray itself, so I reopen the issue.

@alexamici alexamici reopened this Dec 28, 2020
@alexamici alexamici mentioned this issue Dec 28, 2020
Merged
4 tasks
@DanielAdriaansen
Copy link
Contributor Author

Looks like an effective way to fix the bug is in xarray itself, so I reopen the issue.

Thanks so much for the reply and the swift fix! Much appreciated @alexamici and thanks for the hard work getting GRIB engine support into Xarray!

alexamici added a commit that referenced this issue Jan 2, 2021
@alexamici
Declare only BASIC indexing for cfgrib. Fixes #4733 and ecmwf/cfgrib#157
0f1eb96 
 (#4737)

* Declare only BASIC indexing for cfgrib. Fixes #4733 and ecmwf/cfgrib#157

* Add what's news entry.

* Fix RST syntax

* Fix ref to PR
@alexamici alexamici mentioned this issue Jan 3, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic-backends
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Declare only BASIC indexing for cfgrib. Fixes #4733 and ecmwf/cfgrib#157
3 participants
@alexamici @dcherian @DanielAdriaansen