BUG: Dask distributed integration tests failing on Travis #1540

jhamman · 2017-08-31T05:41:50Z

Recent builds on travis are failing for the integration tests for dask distributed (example). Those tests are:

test_dask_distributed_integration_test[h5netcdf]
test_dask_distributed_integration_test[netcdf4]

The traceback includes this detail:

_______________ test_dask_distributed_integration_test[netcdf4] ________________
loop = <tornado.platform.epoll.EPollIOLoop object at 0x7fe36dc9e250>
engine = 'netcdf4'
    @pytest.mark.parametrize('engine', ENGINES)
    def test_dask_distributed_integration_test(loop, engine):
        with cluster() as (s, _):
            with distributed.Client(s['address'], loop=loop):
                original = create_test_data()
                with create_tmp_file(allow_cleanup_failure=ON_WINDOWS) as filename:
                    original.to_netcdf(filename, engine=engine)
                    with xr.open_dataset(filename, chunks=3, engine=engine) as restored:
                        assert isinstance(restored.var1.data, da.Array)
>                       computed = restored.compute()
xarray/tests/test_distributed.py:33: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
xarray/core/dataset.py:487: in compute
    return new.load()
xarray/core/dataset.py:464: in load
    evaluated_data = da.compute(*lazy_data.values())
../../../miniconda/envs/test_env/lib/python2.7/site-packages/dask/base.py:206: in compute
    results = get(dsk, keys, **kwargs)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/client.py:1923: in get
    results = self.gather(packed, asynchronous=asynchronous)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/client.py:1368: in gather
    asynchronous=asynchronous)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/client.py:540: in sync
    return sync(self.loop, func, *args, **kwargs)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/utils.py:239: in sync
    six.reraise(*error[0])
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/utils.py:227: in f
    result[0] = yield make_coro()
../../../miniconda/envs/test_env/lib/python2.7/site-packages/tornado/gen.py:1055: in run
    value = future.result()
../../../miniconda/envs/test_env/lib/python2.7/site-packages/tornado/concurrent.py:238: in result
    raise_exc_info(self._exc_info)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/tornado/gen.py:1063: in run
    yielded = self.gen.throw(*exc_info)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/client.py:1246: in _gather
    traceback)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   c = a[b]
E   TypeError: string indices must be integers

Distributed v.1.18.1 was released 5 days ago so there must have been a breaking change that has been passed down to us.

cc @shoyer, @mrocklin

The text was updated successfully, but these errors were encountered:

shoyer · 2017-08-31T17:20:54Z

Interestingly they only seems to be failing on Python 2.7.

mrocklin · 2017-08-31T18:20:23Z

I was not able to reproduce immediately on my local environment. I'll try to reproduce with the travis-ci environment next time.

…

On Thu, Aug 31, 2017 at 1:21 PM, Stephan Hoyer ***@***.***> wrote: Interestingly they only seems to be failing on Python 2.7. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1540 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszGSLrI9B_pCt-jDD1w8PGPYiI0opks5sdut8gaJpZM4PIVDJ> .

spencerahill · 2017-08-31T18:43:23Z

One bit that may be helpful: this is arising in aospy's tests, e.g. here. But, they're only occurring in our 'py27-min' environment (and are doing so with 100% consistency); never in our full 'py27' environment.

environment-py27-min.yml:

name: test_env
channels:
  - conda-forge
dependencies:
  - python=2.7
  - scipy
  - netCDF4
  - xarray
  - dask
  - distributed
  - pip:
    - coveralls
    - pytest-cov

environment-py27.yml

name: test_env
channels:
  - conda-forge
dependencies:
  - python=2.7
  - scipy
  - netCDF4
  - xarray
  - dask
  - distributed
  - pytest
  - future
  - matplotlib
  - ipython
  - pip:
    - coveralls
    - pytest-cov
    - pytest-catchlog
    - runipy

mrocklin · 2017-08-31T19:14:24Z

I've identified the problem and will have a fix soon. There was another issue with this release so I'll probably issue a micro-release shortly.

…

On Thu, Aug 31, 2017 at 2:43 PM, Spencer Hill ***@***.***> wrote: One bit that may be helpful: this is arising in aospy's tests, e.g. here <https://travis-ci.org/spencerahill/aospy/builds/270528201?utm_source=github_status&utm_medium=notification>. But, they're only occurring in our 'py27-min' environment (and are doing so with 100% consistency); never in our full 'py27' environment. environment-py27-min.yml <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27-min.yml> : name: test_env channels: - conda-forge dependencies: - python=2.7 - scipy - netCDF4 - xarray - dask - distributed - pip: - coveralls - pytest-cov environment-py27.yml <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27.yml> name: test_env channels: - conda-forge dependencies: - python=2.7 - scipy - netCDF4 - xarray - dask - distributed - pytest - future - matplotlib - ipython - pip: - coveralls - pytest-cov - pytest-catchlog - runipy — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1540 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszAePI7b7ThHzz5iQA7k9Ku9roRdpks5sdv7NgaJpZM4PIVDJ> .

mrocklin · 2017-08-31T19:14:44Z

shortly -> within a few days

…

On Thu, Aug 31, 2017 at 3:14 PM, Matthew Rocklin ***@***.***> wrote: I've identified the problem and will have a fix soon. There was another issue with this release so I'll probably issue a micro-release shortly. On Thu, Aug 31, 2017 at 2:43 PM, Spencer Hill ***@***.***> wrote: > One bit that may be helpful: this is arising in aospy's tests, e.g. here > <https://travis-ci.org/spencerahill/aospy/builds/270528201?utm_source=github_status&utm_medium=notification>. > But, they're only occurring in our 'py27-min' environment (and are doing so > with 100% consistency); never in our full 'py27' environment. > > environment-py27-min.yml > <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27-min.yml> > : > > name: test_env > channels: > - conda-forge > dependencies: > - python=2.7 > - scipy > - netCDF4 > - xarray > - dask > - distributed > - pip: > - coveralls > - pytest-cov > > environment-py27.yml > <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27.yml> > > name: test_env > channels: > - conda-forge > dependencies: > - python=2.7 > - scipy > - netCDF4 > - xarray > - dask > - distributed > - pytest > - future > - matplotlib > - ipython > - pip: > - coveralls > - pytest-cov > - pytest-catchlog > - runipy > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1540 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AASszAePI7b7ThHzz5iQA7k9Ku9roRdpks5sdv7NgaJpZM4PIVDJ> > . >

In dask#1328 we started allowing Python 2 to use unicode keys, mostly to support Julia. However this change missed a the pack_data function used in worker.py which still only checked for string-based keys. This arose any time whenever a task depended on a unicode key. Fixes pydata/xarray#1540

mrocklin · 2017-08-31T19:26:41Z

This should be resolved in dask/distributed#1370

mrocklin · 2017-08-31T19:32:48Z

To properly debug this I had to make an async test (most dask.distributed testing does this, search for gen_cluster if curious). This was harder than I would have liked with xarray. We might consider switching some things around to be agnostic to synchronous/asynchronous APIs in the near to moderate future (admittedly, most of this work is probably on me).

…

On Thu, Aug 31, 2017 at 3:14 PM, Matthew Rocklin ***@***.***> wrote: shortly -> within a few days On Thu, Aug 31, 2017 at 3:14 PM, Matthew Rocklin ***@***.***> wrote: > I've identified the problem and will have a fix soon. There was another > issue with this release so I'll probably issue a micro-release shortly. > > On Thu, Aug 31, 2017 at 2:43 PM, Spencer Hill ***@***.***> > wrote: > >> One bit that may be helpful: this is arising in aospy's tests, e.g. here >> <https://travis-ci.org/spencerahill/aospy/builds/270528201?utm_source=github_status&utm_medium=notification>. >> But, they're only occurring in our 'py27-min' environment (and are doing so >> with 100% consistency); never in our full 'py27' environment. >> >> environment-py27-min.yml >> <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27-min.yml> >> : >> >> name: test_env >> channels: >> - conda-forge >> dependencies: >> - python=2.7 >> - scipy >> - netCDF4 >> - xarray >> - dask >> - distributed >> - pip: >> - coveralls >> - pytest-cov >> >> environment-py27.yml >> <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27.yml> >> >> name: test_env >> channels: >> - conda-forge >> dependencies: >> - python=2.7 >> - scipy >> - netCDF4 >> - xarray >> - dask >> - distributed >> - pytest >> - future >> - matplotlib >> - ipython >> - pip: >> - coveralls >> - pytest-cov >> - pytest-catchlog >> - runipy >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#1540 (comment)>, >> or mute the thread >> <https://github.com/notifications/unsubscribe-auth/AASszAePI7b7ThHzz5iQA7k9Ku9roRdpks5sdv7NgaJpZM4PIVDJ> >> . >> > >

In #1328 we started allowing Python 2 to use unicode keys, mostly to support Julia. However this change missed a the pack_data function used in worker.py which still only checked for string-based keys. This arose any time whenever a task depended on a unicode key. Fixes pydata/xarray#1540

mrocklin · 2017-09-01T01:09:52Z

I've merged that PR. If you're testing against git-master then you might want to try retriggering tests.

mrocklin · 2017-09-01T12:19:02Z

I'm surprised that my Fixed #XXXX comment was able to close this issue. Since when am I able to close XArray issues?

shoyer · 2017-09-05T09:18:01Z

@mrocklin I suspect it's because you have admin access to the "pydata" GitHub org

jhamman added topic-dask topic-testing labels Aug 31, 2017

spencerkclark mentioned this issue Aug 31, 2017

Retain original data's mask when yearly averaging spencerahill/aospy#196

Merged

dopplershift mentioned this issue Aug 31, 2017

ENH: Support using opened netCDF4.Dataset (Fixes #1459) #1508

Merged

4 tasks

jhamman added the bug label Aug 31, 2017

mrocklin mentioned this issue Aug 31, 2017

Support unicode keys in pack_data dask/distributed#1370

Merged

mrocklin closed this as completed in dask/distributed#1370 Sep 1, 2017

shoyer mentioned this issue Sep 1, 2017

Add pathlib.Path support to open_(mf)dataset #1514

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Dask distributed integration tests failing on Travis #1540

BUG: Dask distributed integration tests failing on Travis #1540

jhamman commented Aug 31, 2017 •

edited

Loading

shoyer commented Aug 31, 2017

mrocklin commented Aug 31, 2017 via email

spencerahill commented Aug 31, 2017

mrocklin commented Aug 31, 2017 via email

mrocklin commented Aug 31, 2017 via email

mrocklin commented Aug 31, 2017

mrocklin commented Aug 31, 2017 via email

mrocklin commented Sep 1, 2017

mrocklin commented Sep 1, 2017

shoyer commented Sep 5, 2017

BUG: Dask distributed integration tests failing on Travis #1540

BUG: Dask distributed integration tests failing on Travis #1540

Comments

jhamman commented Aug 31, 2017 • edited Loading

shoyer commented Aug 31, 2017

mrocklin commented Aug 31, 2017 via email

spencerahill commented Aug 31, 2017

mrocklin commented Aug 31, 2017 via email

mrocklin commented Aug 31, 2017 via email

mrocklin commented Aug 31, 2017

mrocklin commented Aug 31, 2017 via email

mrocklin commented Sep 1, 2017

mrocklin commented Sep 1, 2017

shoyer commented Sep 5, 2017

jhamman commented Aug 31, 2017 •

edited

Loading