Skip to content

INTERNALERROR> IOError: cannot send (already closed?) #2223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
godfryd opened this issue Jan 30, 2017 · 6 comments
Closed

INTERNALERROR> IOError: cannot send (already closed?) #2223

godfryd opened this issue Jan 30, 2017 · 6 comments

Comments

@godfryd
Copy link

godfryd commented Jan 30, 2017 •
edited by RonnyPfannschmidt
Loading

Hello,

I'm using pytest with timeout, xdist and coverage plugins.
I quite often gets internal errors like the one below.
Do you have any tips how to deal with that?

========================================================================================= test session starts ==========================================================================================
platform linux2 -- Python 2.7.12, pytest-3.0.6, py-1.4.32, pluggy-0.4.0 -- /home/godfryd/venv/bin/python2
cachedir: .cache
rootdir: /home/godfryd/tests, inifile:
plugins: xdist-1.15.0, timeout-1.2.0, cov-2.4.0
timeout: 3600.0s method: signal
[gw0] linux2 Python 2.7.12 cwd: /home/godfryd/tests
[gw1] linux2 Python 2.7.12 cwd: /home/godfryd/tests
[gw2] linux2 Python 2.7.12 cwd: /home/godfryd/tests
[gw3] linux2 Python 2.7.12 cwd: /home/godfryd/tests
[gw0] Python 2.7.12 (default, Jul  1 2016, 15:12:24)  -- [GCC 5.4.0 20160609]
[gw1] Python 2.7.12 (default, Jul  1 2016, 15:12:24)  -- [GCC 5.4.0 20160609]
[gw2] Python 2.7.12 (default, Jul  1 2016, 15:12:24)  -- [GCC 5.4.0 20160609]
[gw3] Python 2.7.12 (default, Jul  1 2016, 15:12:24)  -- [GCC 5.4.0 20160609]
gw0 [158] / gw1 [158] / gw2 [158] / gw3 [158]
scheduling tests via LoadScheduling
....
 INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 133, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 61, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 77, in run_tests
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 245, in _wrapped_call
INTERNALERROR>     next(wrap_controller)   # first yield
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/pytest_timeout.py", line 76, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/pytest_timeout.py", line 115, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR> ValueError: signal only works in main thread

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 133, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 536, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 555, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 593, in slave_slavefinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('test_abn.py::TestWithPerformancePreprocessor::test_with_instability', <SlaveController gw3>)
INTERNALERROR> assert not 'test_abn.py::TestWithPerformancePreprocessor::test_with_instability'

===================================================================================== no tests ran in 1.10 seconds =====================================================================================
Makefile:366: recipe for target 'tests' failed
make: *** [tests] Error 3
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 133, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 61, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 77, in run_tests
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 280, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 280, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/runner.py", line 66, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/runner.py", line 79, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/runner.py", line 137, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 92, in pytest_runtest_logreport
INTERNALERROR>   File "<remote exec>", line 25, in sendevent
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 717, in send
INTERNALERROR>     self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 990, in _send
INTERNALERROR>     raise IOError("cannot send (already closed?)")
INTERNALERROR> IOError: cannot send (already closed?)
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 133, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 61, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 77, in run_tests
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 280, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 280, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/runner.py", line 66, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/runner.py", line 79, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/runner.py", line 137, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 92, in pytest_runtest_logreport
INTERNALERROR>   File "<remote exec>", line 25, in sendevent
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 717, in send
INTERNALERROR>     self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR>   File "/home/godfryd/venv/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 990, in _send
INTERNALERROR>     raise IOError("cannot send (already closed?)")
INTERNALERROR> IOError: cannot send (already closed?)
@nicoddemus
Copy link
Member

Usually internal errors like this might happen because of a crash in one of the slaves... other than that it is hard to diagnose what is the problem. :/

@godfryd
Copy link
Author

godfryd commented Jan 30, 2017

how to learn more about such crash?

@nicoddemus
Copy link
Member

Does it happen when you execute without xdist?

Using catchsegv might also be useful.

@godfryd
Copy link
Author

godfryd commented Jan 30, 2017

it happens only with xdist

@RonnyPfannschmidt
Copy link
Member

pytest-timeout seems to be incompatible with xdist and doesn't warn about that

@hpk42 i believe the recent execmodel change of execnet is to blame (it manages the thread model and handling differently, however i yet have to test this theory)

@RonnyPfannschmidt
Copy link
Member

closing this one as issue with pytest-xdist and pytest_timeout - there is nothing pytest itself can do about it

@ghost ghost mentioned this issue May 8, 2017
Closed
@gfyoung gfyoung mentioned this issue Mar 3, 2018
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@RonnyPfannschmidt @godfryd @nicoddemus