Skip to content

Commit ccb31f7

Browse files
committed
Update threads per worker to 1 in configs and docs
This is done to increase the reliability of Reshapr extractions. With threads>1 we see random occurrences of errors like: `RuntimeError: NetCDF: Not a valid ID` The root cause appears to be that the `netcdf-c` is not thread-safe and a change introduced in `netcdf4-python=1.6.1` removed a work-around for the lack of thread-safety. See discussion in xCDAT/xcdat#561 and in other discussions linked there.
1 parent 5a600e1 commit ccb31f7

File tree

5 files changed

+5
-5
lines changed

5 files changed

+5
-5
lines changed

cluster_configs/salish_cluster.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,5 @@
33
name: salish dask cluster
44
processes: True
55
number of workers: 4
6-
threads per worker: 4
6+
threads per worker: 1
77
memory limit: auto

docs/dask_clusters.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ a ``bash`` loop to resample day-averaged datasets to get month-averaged datasets
9999
.. code-block:: bash
100100
101101
$ conda activate reshapr
102-
(reshapr)$ dask-worker --nworkers=1 --nthreads=4 142.103.36.12:8786 &
102+
(reshapr)$ dask-worker --nworkers=1 --nthreads=1 142.103.36.12:8786 &
103103
104104
Use :kbd:`Control-b ,` to rename the ``tmux`` terminal to ``dask-workers``.
105105

docs/subcommands/info.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ Example:
9090
name: salish dask cluster
9191
processes: True
9292
number of workers: 4
93-
threads per worker: 4
93+
threads per worker: 1
9494
9595
Please use reshapr info --help to learn how to get other information,
9696
or reshapr --help to learn about other sub-commands.

tests/core/test_info.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ def test_cluster_file_contents(self, capsys):
123123
name: salish dask cluster
124124
processes: True
125125
number of workers: 4
126-
threads per worker: 4
126+
threads per worker: 1
127127
"""
128128
).splitlines()
129129
assert [line.strip() for line in stdout_lines[1:7]] == expected

tests/test_cluster_configs.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ def test_salish_cluster(self):
5757
assert cluster_config["name"] == "salish dask cluster"
5858
assert cluster_config["processes"] is True
5959
assert cluster_config["number of workers"] == 4
60-
assert cluster_config["threads per worker"] == 4
60+
assert cluster_config["threads per worker"] == 1
6161
assert cluster_config["memory limit"] == "auto"
6262

6363

0 commit comments

Comments
 (0)