Problems opening ROMS restart format files

[dhruvbalwada]

Hello,

I am trying to open some netcdf files which were generated by ROMS. However, I keep the getting the following error. Any help would be appreciated.

I am able to open other netcdf files.

The xarray version is 0.9.1

The header to the netcdf file is here-
https://hackpad.com/header-O8RfdHcunnX

The error I get is
ValueErrorTraceback (most recent call last)
in ()
----> 1 ds_flux = xr.open_mfdataset(os.path.join(ddir, 'PHY_FLUXES_FILES/*.nc'))
2 ds_flux
3
4 xr.version

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/api.pyc in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, lock, **kwargs)
503 lock = _default_lock(paths[0], engine)
504 datasets = [open_dataset(p, engine=engine, chunks=chunks or {}, lock=lock,
--> 505 **kwargs) for p in paths]
506 file_objs = [ds._file_obj for ds in datasets]
507

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/api.pyc in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables)
287 store = backends.NetCDF4DataStore(filename_or_obj, group=group)
288 elif engine == 'scipy':
--> 289 store = backends.ScipyDataStore(filename_or_obj)
290 elif engine == 'pydap':
291 store = backends.PydapDataStore(filename_or_obj)

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/scipy_.pyc in init(self, filename_or_obj, mode, format, group, writer, mmap)
100 filename=filename_or_obj,
101 mode=mode, mmap=mmap, version=version)
--> 102 self.ds = opener()
103 self._opener = opener
104 self._mode = mode

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/scipy_.pyc in _open_scipy_netcdf(filename, mode, mmap, version)
63
64 return scipy.io.netcdf_file(filename, mode=mode, mmap=mmap,
---> 65 version=version)
66
67

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/scipy/io/netcdf.pyc in init(self, filename, mode, mmap, version, maskandscale)
262
263 if mode in 'ra':
--> 264 self._read()
265
266 def setattr(self, attr, value):

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/scipy/io/netcdf.pyc in _read(self)
584 self._read_dim_array()
585 self._read_gatt_array()
--> 586 self._read_var_array()
587
588 def _read_numrecs(self):

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/scipy/io/netcdf.pyc in _read_var_array(self)
689 # Build rec array.
690 if self.use_mmap:
--> 691 rec_array = self._mm_buf[begin:begin+self._recs*self._recsize].view(dtype=dtypes)
692 rec_array.shape = (self._recs,)
693 else:

ValueError: new type not compatible with array.

[fmaussion]

Could you try these two things, too?

• install NetCDF4
• open a single file with open_dataset instead of open_mfdataset

[dhruvbalwada]

I already had netcdf4 installed. I am able to open other netcdf files.

I get a similar error when I use open_dataset --

Exception AttributeError: "'netcdf_file' object has no attribute 'fp'" in <bound method netcdf_file.close of <scipy.io.netcdf.netcdf_file object at 0x7eda808e8150>> ignored

ValueErrorTraceback (most recent call last)
in ()
----> 1 ds_flux = xr.open_dataset(os.path.join(ddir, 'PHY_FLUXES_FILES/roms_dia_avg_y17.nc'))
2 ds_flux

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/api.pyc in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables)
287 store = backends.NetCDF4DataStore(filename_or_obj, group=group)
288 elif engine == 'scipy':
--> 289 store = backends.ScipyDataStore(filename_or_obj)
290 elif engine == 'pydap':
291 store = backends.PydapDataStore(filename_or_obj)

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/scipy_.pyc in init(self, filename_or_obj, mode, format, group, writer, mmap)
100 filename=filename_or_obj,
101 mode=mode, mmap=mmap, version=version)
--> 102 self.ds = opener()
103 self._opener = opener
104 self._mode = mode

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/scipy_.pyc in _open_scipy_netcdf(filename, mode, mmap, version)
63
64 return scipy.io.netcdf_file(filename, mode=mode, mmap=mmap,
---> 65 version=version)
66
67

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/scipy/io/netcdf.pyc in init(self, filename, mode, mmap, version, maskandscale)
262
263 if mode in 'ra':
--> 264 self._read()
265
266 def setattr(self, attr, value):

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/scipy/io/netcdf.pyc in _read(self)
584 self._read_dim_array()
585 self._read_gatt_array()
--> 586 self._read_var_array()
587
588 def _read_numrecs(self):

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/scipy/io/netcdf.pyc in _read_var_array(self)
689 # Build rec array.
690 if self.use_mmap:
--> 691 rec_array = self._mm_buf[begin:begin+self._recs*self._recsize].view(dtype=dtypes)
692 rec_array.shape = (self._recs,)
693 else:

ValueError: new type not compatible with array.

[fmaussion]

For some reason the bug comes form the scipy engine. What happens if you do:

xr.open_dataset(os.path.join(ddir, 'PHY_FLUXES_FILES/roms_dia_avg_y17.nc'), engine='netcdf4')

?

Could you send us one of your files?

[dhruvbalwada]

Working on sharing the file now.

I get this error when I tried
xr.open_dataset(os.path.join(ddir, 'PHY_FLUXES_FILES/roms_dia_avg_y17.nc'), engine='netcdf4')

ImportErrorTraceback (most recent call last)
in ()
----> 1 xr.open_dataset(os.path.join(ddir, 'PHY_FLUXES_FILES/roms_dia_avg_y17.nc'), engine='netcdf4')

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/api.pyc in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables)
285 allow_remote=True)
286 if engine == 'netcdf4':
--> 287 store = backends.NetCDF4DataStore(filename_or_obj, group=group)
288 elif engine == 'scipy':
289 store = backends.ScipyDataStore(filename_or_obj)

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in init(self, filename, mode, format, group, writer, clobber, diskless, persist)
203 diskless=diskless, persist=persist,
204 format=format)
--> 205 self.ds = opener()
206 self.format = format
207 self.is_remote = is_remote_uri(filename)

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in _open_netcdf4_group(filename, mode, group, **kwargs)
177
178 def _open_netcdf4_group(filename, mode, group=None, **kwargs):
--> 179 import netCDF4 as nc4
180
181 ds = nc4.Dataset(filename, mode=mode, **kwargs)

/home/dbalwada/.conda/envs/dhruvenv/lib/python2.7/site-packages/netCDF4/init.py in ()
1 # init for netCDF4. package
2 # Docstring comes from extension module _netCDF4.
----> 3 from ._netCDF4 import *
4 # Need explicit imports for names beginning with underscores
5 from ._netCDF4 import doc, pdoc

ImportError: libmfhdf.so.0: cannot open shared object file: No such file or directory

[fmaussion]

The later error says that your NetCDF4 library isn't properly installed. But the file problem might be unrelated, I'd need to have a look at the file.

[rabernat]

@dhruvbalwada, if you can't import netCDF4, you need to look at your conda environment and make sure all the necessary packages are installed. Your error indicates some sort of library dependency issue which should hypothetically have been resolved by conda. Here is the environment used for testing xarray with python 2.7.

[dhruvbalwada]

Thanks Ryan. This problem has now been resolved.
I did have netcdf4 but was missing hdf4, which is the reason that netcdf4 module won't load. I am a little surprised, if hdf4 is a dependency (lack of which doesn't even allow the module to load), then why is not automatically installed when conda installs netcdf4?
A similar issue was raised here -
Unidata/netcdf4-python#574