Skip to content

Flaky tests in Python 3.5.0 with xdist 😬 #5795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nicoddemus opened this issue Aug 28, 2019 · 1 comment · Fixed by #5824
Closed

Flaky tests in Python 3.5.0 with xdist 😬 #5795

nicoddemus opened this issue Aug 28, 2019 · 1 comment · Fixed by #5824
Labels
type: infrastructure improvement to development/releases/CI structure

Comments

@nicoddemus
Copy link
Member

nicoddemus commented Aug 28, 2019 •
edited
Loading 

__________________________ testing/test_collection.py __________________________
[gw1] linux -- Python 3.5.0 /home/travis/build/pytest-dev/pytest/.tox/py35-xdist/bin/python
worker 'gw1' crashed while running 'testing/test_collection.py::test_exit_on_collection_with_maxfail_bigger_than_n_errors'

https://travis-ci.org/pytest-dev/pytest/jobs/577770604

😬

In #5803 (comment), we suspect this is related to 3.5.0, as we have never seen this failure before started testing on that Python version on Travis.
@nicoddemus nicoddemus added the type: infrastructure improvement to development/releases/CI structure label Aug 28, 2019
@nicoddemus nicoddemus mentioned this issue Aug 28, 2019
Merged
4 tasks
nicoddemus added a commit to nicoddemus/pytest that referenced this issue Aug 30, 2019
@nicoddemus
Mark flaky test as xfail
5329128 
Related to pytest-dev#5795
@nicoddemus nicoddemus mentioned this issue Aug 30, 2019
Merged
nicoddemus added a commit to nicoddemus/pytest that referenced this issue Aug 30, 2019
@nicoddemus
Mark flaky test as xfail on 3.5.0
11b8405 
Ref: pytest-dev#5795
@nicoddemus nicoddemus changed the title test_exit_on_collection_with_maxfail_bigger_than_n_errors flaky in py35 😬 test_exit_on_collection_with_maxfail_bigger_than_n_errors flaky in Python 3.5.0 😬 Aug 30, 2019
nicoddemus added a commit to nicoddemus/pytest that referenced this issue Aug 30, 2019
@nicoddemus
Run py35 without xdist on Travis
bb60736 
Due to the flaky tests in 3.5.0, drop running
py35 with xdist for now in the hope we get better error
messages.

Ref: pytest-dev#5795
@nicoddemus nicoddemus changed the title test_exit_on_collection_with_maxfail_bigger_than_n_errors flaky in Python 3.5.0 😬 Flaky tests in Python 3.5.0 with xdist 😬 Aug 30, 2019
@nicoddemus nicoddemus mentioned this issue Aug 30, 2019
Merged
@nicoddemus
Copy link
Member Author

Unfortunately py35, without xdist and coverage, still is crashing randomly.

Unless someone has other ideas, I think we should consider not testing Python 3.5.0 anymore. Having it failing randomly is too disruptive.

Does flake8 now cover most gotchas related to poor typing in 3.5.0?

blueyed added a commit to blueyed/pytest that referenced this issue Sep 6, 2019
@blueyed
ci: Travis: do not test with 3.5.0
f1b605c 
This causes flaky test failures (crashes).

Closes pytest-dev#5795.
@blueyed blueyed mentioned this issue Sep 6, 2019
Merged
blueyed added a commit to blueyed/pytest that referenced this issue Jan 10, 2020
@blueyed
ci: Travis: Python 3.5.1 via Trusty
d67526b 
Python 3.5.0 caused flaky failures before
(pytest-dev#5795), hopefully this is
more stable.

This is pulled out of pytest-dev#6435,
which adds code specific for Python < 3.5.2, and therefore it makes
sense to test it.
@blueyed blueyed mentioned this issue Jan 10, 2020
Merged
blueyed added a commit to blueyed/pytest that referenced this issue Jan 11, 2020
@blueyed
ci: Travis: Python 3.5.1 via Trusty
ead6222 
Python 3.5.0 caused flaky failures before
(pytest-dev#5795), hopefully this is
more stable.

This is pulled out of pytest-dev#6435,
which adds code specific for Python < 3.5.2, and therefore it makes
sense to test it.
blueyed added a commit to blueyed/pytest that referenced this issue Jan 12, 2020
@blueyed
ci: Travis: Python 3.5.1 via Trusty
f95ae80 
Python 3.5.0 caused flaky failures before
(pytest-dev#5795).

This is pulled out of pytest-dev#6435,
which adds code specific for Python < 3.5.2.

It only runs a specific test, while collecting everything to get
coverage of the version specific code around typing.
blueyed added a commit to blueyed/pytest that referenced this issue Jan 12, 2020
@blueyed
ci: Travis: Python 3.5.1 via Trusty
b784280 
Python 3.5.0 caused flaky failures before
(pytest-dev#5795).

This is pulled out of pytest-dev#6435,
which adds code specific for Python < 3.5.2.

It only runs a specific test, while collecting everything to get
coverage of the version specific code around typing.
blueyed added a commit to blueyed/pytest that referenced this issue Jan 14, 2020
@blueyed
ci: Travis: Python 3.5.1 via Trusty
4ff7453 
Python 3.5.0 caused flaky failures before
(pytest-dev#5795).

This is pulled out of pytest-dev#6435,
which adds code specific for Python < 3.5.2.

It only runs a specific test, while collecting everything to get
coverage of the version specific code around typing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: infrastructure improvement to development/releases/CI structure
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ci: Travis: do not test with 3.5.0 blueyed/pytest
1 participant
@nicoddemus