New to Python and Jupiter

[sandrocalmanti]

I'm new to python and to using notebooks so this might be an easy one.

Just to get familiar with using jupyter notebooks, I'm trying to run the template for computing global monthly time series:
https://github.com/bopen/c3s-eqc-toolbox-template/blob/main/notebooks/climate_projections/01-Application_Template_Global_Timeseries_Monthly.ipynb

It seems I have a problem with download.py which I'm not able to handle. You can find my test notebook here?

/data/wp3/calmanti_sandro/c3s-eqc-toolbox-template/notebooks/climate_projections/my.ipynb

I've followed the instruction provided today during the training. Is it possible that I'm missing something in configuring the environment or similar?

Thanks for helping

[malmans2]

Hi @sandrocalmanti,

Sorry, that's actually my fault. I introduced a bug today while working on an issue for wp5.
I've updated all environments and your notebook should work now on the VM (remember to restart the kernel before running the notebook again).

You'll probably run into a different error. Looks like whoever added the .cdsapirc for wp3 did not approve the terms and conditions for cmip6. So you need to use your own cdsapirc. I briefly mentioned it today. See the first cell in this notebook or this comment.

[sandrocalmanti]

Thank you @malmans2 ,

I see there can be an issue with the .cdsapirc for some datasers.

Indeed some of the notebooks are working now (e.g. the examples on seasonal forecasts). On the other hand, I've tried using the scanrios after substituting my .cdsapirc and after approving the terms and conditions but it's still not working. It might need some time to process the changes, so I'll try later with this.

I'm still getting another problem which doesn't seems to be related to terms&condition. For example, while this notebook works fine (although it doesn't show any graphical output on screen, just the file)

02-Application_Template_Global_Timeseries_Pressure_Levels.zip

the next one ERA5 data as well but it gives an error related to the processing function.

01-Application_Template_Global_Timeseries_Single_Level.zip

UnboundLocalError: local variable 'preprocess' referenced before assignment

Is it related to the environment I'm using?

Cheers

Sandro

[malmans2]

the next one ERA5 data as well but it gives an error related to the processing function.

01-Application_Template_Global_Timeseries_Single_Level.zip

UnboundLocalError: local variable 'preprocess' referenced before assignment

The UnboundLocalError should be fixed in the virtual machine environment.
Try to restart the kernel and re-run the notebook.

I'm running it right now on the VM and I don't see that error. I see there are connection errors, but nothing to do with the UnboundLocalError. (Unfortunately, I can't do much with those connections errors on the VM. See: #14)

If you are running on a local machine, you need to update the environment.

[malmans2]

Indeed some of the notebooks are working now (e.g. the examples on seasonal forecasts). On the other hand, I've tried using the scanrios after substituting my .cdsapirc and after approving the terms and conditions but it's still not working. It might need some time to process the changes, so I'll try later with this.

Yesterday it was very quick for me, but who knows... Make sure you point to the right cdsapirc on top of your notebook, before importing any other package.
I.e.:

import os
os.environ["CDSAPI_RC"] = os.path.expanduser("~/calmanti_sandro/.cdsapirc")

from c3s_eqc_automatic_quality_control import diagnostics, download
...

[virginiaciardini]

Hi @malmans2,
I'm trying to run the template for Global_Timeseries_Ungridded; I firsty resolve the issue with the .cdsapirc but now after run "Plot and save figure" I receive the following error message:

KeyError Traceback (most recent call last)
Cell In[11], line 3
1 filename = "".join(["global-timeseries", collection_id])
2 title = filename.replace("-", " ").replace("", " ").title()
----> 3 fig = plot.shaded_std(
4 list(ds.data_vars),
5 ds.sel(reduction="mean"),
6 title=title,
7 )
8 fig.show()
9 fig.write_image(filename + ".png")

File /data/common/mambaforge/envs/wp5/lib/python3.10/site-packages/c3s_eqc_automatic_quality_control/plot.py:107, in shaded_std(vars, ds_mean, ds_std, hue_dim, title, x_dim)
104 if not isinstance(rgb, tuple):
105 rgb = pc.hex_to_rgb(rgb)
--> 107 da_mean = mean[var].where(mean[var].notnull(), drop=True).squeeze()
108 if da_mean.size <= 1:
109 continue

File /data/common/mambaforge/envs/wp5/lib/python3.10/site-packages/xarray/core/common.py:1141, in DataWithCoords.where(self, cond, other, drop)
1138 for dim in cond.sizes.keys():
1139 indexers[dim] = _get_indexer(dim)
-> 1141 self = self.isel(**indexers)
1142 cond = cond.isel(**indexers)
1144 return ops.where_method(self, cond, other)

File /data/common/mambaforge/envs/wp5/lib/python3.10/site-packages/xarray/core/dataarray.py:1387, in DataArray.isel(self, indexers, drop, missing_dims, **indexers_kwargs)
1384 indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "isel")
1386 if any(is_fancy_indexer(idx) for idx in indexers.values()):
-> 1387 ds = self._to_temp_dataset()._isel_fancy(
1388 indexers, drop=drop, missing_dims=missing_dims
1389 )
1390 return self._from_temp_dataset(ds)
1392 # Much faster algorithm for when all indexers are ints, slices, one-dimensional
1393 # lists, or zero or one-dimensional np.ndarray's

File /data/common/mambaforge/envs/wp5/lib/python3.10/site-packages/xarray/core/dataset.py:2483, in Dataset._isel_fancy(self, indexers, drop, missing_dims)
2479 var_indexers = {
2480 k: v for k, v in valid_indexers.items() if k in var.dims
2481 }
2482 if var_indexers:
-> 2483 new_var = var.isel(indexers=var_indexers)
2484 # drop scalar coordinates
2485 # pydata/xarray#6554
2486 if name in self.coords and drop and new_var.ndim == 0:

File /data/common/mambaforge/envs/wp5/lib/python3.10/site-packages/xarray/core/variable.py:1328, in Variable.isel(self, indexers, missing_dims, **indexers_kwargs)
1325 indexers = drop_dims_from_indexers(indexers, self.dims, missing_dims)
1327 key = tuple(indexers.get(dim, slice(None)) for dim in self.dims)
-> 1328 return self[key]

File /data/common/mambaforge/envs/wp5/lib/python3.10/site-packages/xarray/core/variable.py:875, in Variable.getitem(self, key)
862 def getitem(self: T_Variable, key) -> T_Variable:
863 """Return a new Variable object whose contents are consistent with
864 getting the provided key from the underlying data.
865
(...)
873 array x.values directly.
874 """
--> 875 dims, indexer, new_order = self._broadcast_indexes(key)
876 data = as_indexable(self._data)[indexer]
877 if new_order:

File /data/common/mambaforge/envs/wp5/lib/python3.10/site-packages/xarray/core/variable.py:708, in Variable._broadcast_indexes(self, key)
705 if all(isinstance(k, BASIC_INDEXING_TYPES) for k in key):
706 return self._broadcast_indexes_basic(key)
--> 708 self._validate_indexers(key)
709 # Detect it can be mapped as an outer indexer
710 # If all key is unlabeled, or
711 # key can be mapped as an OuterIndexer.
712 if all(not isinstance(k, Variable) for k in key):

File /data/common/mambaforge/envs/wp5/lib/python3.10/site-packages/xarray/core/variable.py:760, in Variable._validate_indexers(self, key)
755 raise IndexError(
756 "{}-dimensional boolean indexing is "
757 "not supported. ".format(k.ndim)
758 )
759 if is_duck_dask_array(k.data):
--> 760 raise KeyError(
761 "Indexing with a boolean dask array is not allowed. "
762 "This will result in a dask array of unknown shape. "
763 "Such arrays are unsupported by Xarray."
764 "Please compute the indexer first using .compute()"
765 )
766 if getattr(k, "dims", (dim,)) != (dim,):
767 raise IndexError(
768 "Boolean indexer should be unlabeled or on the "
769 "same dimension to the indexed array. Indexer is "
(...)
772 )
773 )

KeyError: 'Indexing with a boolean dask array is not allowed. This will result in a dask array of unknown shape. Such arrays are unsupported by Xarray.Please compute the indexer first using .compute()'

Could you give me some hints to solve?
thanks

[malmans2]

Hi @virginiaciardini,
Good catch, it was a breaking change in the latest version of xarray.
It should be fixed now. I've already updated the VM environment, so all you have to do is to restart the kernel and re-run the notebook.

(PS, Next time please open a new issue, it makes it easier for us to monitor progress)

[malmans2]

Closing this as it should be fixed.
As always, feel free to re-open!