PyGMT load_earth_relief breaks with xarray 0.18.0 (ValueError: cannot guess the engine)

[seisman]

Description of the problem

xarray 0.18.0 was released on May 6, 2021, and some PyGMT tests break. I'm not sure if it's a bug or a breaking change of xarray.

Full code that generated the error

After upgrading to xarray 0.18.0, following codes no longer work:

import pygmt
pygmt.datasets.load_earth_relief("01d")

Full error message

ValueError                                Traceback (most recent call last)
<ipython-input-2-30e5a4024c6c> in <module>
----> 1 pygmt.datasets.load_earth_relief("01d")

pygmt/pygmt/helpers/decorators.py in new_module(*args, **kwargs)
    436                         kwargs[arg] = separators[fmt].join(f"{item}" for item in value)
    437             # Execute the original function and return its output
--> 438             return module_func(*args, **kwargs)
    439
    440         return new_module

pygmt/pygmt/datasets/earth_relief.py in load_earth_relief(resolution, region, registration, use_srtm)
    134             )
    135         fname = which(f"@earth_relief_{resolution}{reg}", download="a")
--> 136         with xr.open_dataarray(fname) as dataarray:
    137             grid = dataarray.load()
    138             _ = grid.gmt  # load GMTDataArray accessor information

~/.miniconda/envs/pygmt/lib/python3.9/site-packages/xarray/backends/api.py in open_dataarray(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, *args, **kwargs)
    657         )
    658
--> 659     dataset = open_dataset(
    660         filename_or_obj,
    661         decode_cf=decode_cf,

~/.miniconda/envs/pygmt/lib/python3.9/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, *args, **kwargs)
    483
    484     if engine is None:
--> 485         engine = plugins.guess_engine(filename_or_obj)
    486
    487     backend = plugins.get_backend(engine)

~/.miniconda/envs/pygmt/lib/python3.9/site-packages/xarray/backends/plugins.py in guess_engine(store_spec)
    110             warnings.warn(f"{engine!r} fails while guessing", RuntimeWarning)
    111
--> 112     raise ValueError("cannot guess the engine, try passing one explicitly")
    113
    114

ValueError: cannot guess the engine, try passing one explicitly

System information

Please paste the output of python -c "import pygmt; pygmt.show_versions()":

PyGMT information:
  version: v0.3.2.dev82+g82ebbbc4.d20210412
System information:
  python: 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:20)  [Clang 11.0.1 ]
  executable: ~/.miniconda/envs/pygmt/bin/python
  machine: macOS-11.2.3-x86_64-i386-64bit
Dependency information:
  numpy: 1.20.1
  pandas: 1.2.3
  xarray: 0.18.0
  netCDF4: 1.5.6
  packaging: 20.9
  ghostscript: 9.53.3
  gmt: 6.1.1

[seisman]

Using xarray directly gives the same error:

import xarray as xr
# Fail with the same error
xr.open_dataarray("~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd")
# Specifying engine="netcdf4" works
xr.open_dataarray("~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd", engine="netcdf4")

[weiji14]

Oops, a bit too slow, I'll close down #1263 as a duplicate.

[seisman]

I'm not familiar with xarray, but I guess that xarray tries to open the grid file using different engines, and it should work when it tries the "netcdf4" engine. It looks like an upstream bug to me, rather than a breaking API change.

[weiji14]

I'm not familiar with xarray, but I guess that xarray tries to open the grid file using different engines, and it should work when it tries the "netcdf4" engine. It looks like an upstream bug to me, rather than a breaking API change.

xarray had a big backend engine refactor to handle multiple file formats (e.g. NetCDF, Zarr, etc) in a plugin-like way. I can try to raise an issue with upstream xarray, but they have a relatively slow release cycle (speaking from experience) and an xarray v0.18.1 probably won't come anytime soon in the next few weeks.

I'm aware that this will cause issues for any PyGMT users creating a new environment with xarray=0.18 and pygmt=0.3.1 installed (at least those using the load_earth_relief function). So, we should probably fix this in PyGMT by explicitly setting the engine. I'm also wondering if this bugfix is critical enough to warrant a quick PyGMT v0.3.2 release.

[seisman]

I can try to raise an issue with upstream xarray

Yes, please.

I'm also wondering if this bugfix is critical enough to warrant a quick PyGMT v0.3.2 release.

You mean creating a branch from the v0.3.1 tag, backport the changes in #1264, and make the v0.3.2 release?

[weiji14]

I can try to raise an issue with upstream xarray

Yes, please.

Is a *.grd file the same as a NetCDF4 file? I've stepped through the xarray code and the easiest fix might be to update the line at https://github.com/pydata/xarray/blob/v0.18.0/xarray/backends/netCDF4_.py#L524), changing ext in {".nc", ".nc4", ".cdf"} to ext in {".nc", ".nc4", ".cdf", ".grd"}, but I'm not sure if it's a good idea (afraid that it might break other libraries).

I'm also wondering if this bugfix is critical enough to warrant a quick PyGMT v0.3.2 release.

You mean creating a branch from the v0.3.1 tag, backport the changes in #1264, and make the v0.3.2 release?

Yes, that is one option (though not sure if I like that idea). Alternatively we can just start working on the PyGMT v0.4.0 release 🙈

Technically we've fixed this issue on the PyGMT side in #1264, but let's keep this issue pinned for a while, and keep watch to see if users make bug reports on the forum/github. If there's a lot of bug reports, maybe then go with a v0.3.2 release.

[seisman]

Related upstream issue reports:

• Engine is no longer inferred for filenames not ending in ".nc" pydata/xarray#5295
• ds = xr.tutorial.load_dataset("air_temperature") with 0.18 needs engine argument pydata/xarray#5291

[seisman]

xarray v0.18.1 has been released (https://github.com/pydata/xarray/releases/tag/v0.18.1). The patch release should fix the problem.