Second test fails when run with first, but passes when run seperately.

[RobRoseKnows]

Currently working on a large project and I'm coming across a weird issue when trying to run my pytests. I created a Gist here that recreates the error I'm having.

I don't know if this is deliberate behavior or if I messed something up but when I run the two tests together (either as a file, or as a project wide pytest), the second test fails because it seems to preserve the same copy of an object from the previous test. The class being tested is a wrapper around a class with some non-implemented features with the ones already implemented being tested. The stuff being tested is iterators and subscripting.

This remains the same if I order them differently in the file, also if I change the names. Not too sure what's going on here, so I don't know what else I can tell you.

Here's the output:

$ pytest -v
================================================ test session starts ================================================
platform linux -- Python 3.6.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0 -- /home/robrose/.pyenv/versions/3.6.5/envs/general/bin/python
cachedir: .pytest_cache
rootdir: /mnt/c/Users/lt-rrose/Dev/debug, inifile:
collected 2 items

test_base.py::test_iter_on_mock_profile PASSED                                                                [ 50%]
test_base.py::test_subscripting_on_mock_profile FAILED                                                        [100%]

===================================================== FAILURES ======================================================
_________________________________________ test_subscripting_on_mock_profile _________________________________________

    @pytest.mark.pytest_style_tests
    def test_subscripting_on_mock_profile():
        mock = ProfileMock([ProviderType.COMPUTE])
        count = 0
        for _ in mock:
            count += 1
>       assert count == 1
E       assert 2 == 1

test_base.py:37: AssertionError
----------------------------------------------- Captured stdout call ------------------------------------------------
ProviderType.COMPUTE
======================================== 1 failed, 1 passed in 0.08 seconds =========================================
$ pytest test_base.py::test_iter_on_mock_profile
================================================ test session starts ================================================
platform linux -- Python 3.6.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /mnt/c/Users/lt-rrose/Dev/debug, inifile:
collected 1 item

test_base.py .                                                                                                [100%]

============================================= 1 passed in 0.03 seconds ==============================================
$ pytest test_base.py::test_subscripting_on_mock_profile
================================================ test session starts ================================================
platform linux -- Python 3.6.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /mnt/c/Users/lt-rrose/Dev/debug, inifile:
collected 1 item

test_base.py .                                                                                                [100%]

============================================= 1 passed in 0.04 seconds ==============================================

pip list output of the fresh pyenv it's running in:

atomicwrites (1.1.5)
attrs (18.1.0)
more-itertools (4.2.0)
pip (9.0.3)
pluggy (0.6.0)
py (1.5.4)
pytest (3.6.3)
setuptools (39.0.1)
six (1.11.0)

OS is Ubuntu 16.04 on Windows Subsystem for Linux.

Here's a quick checklist in what to include:

• Include a detailed description of the bug or suggestion
• pip list of the virtual environment you are using
• pytest and operating system versions
• Minimal example if possible

[pytestbot]

GitMate.io thinks possibly related issues are #1591 (pytest-xdist fails when running same tests several times), #747 (flakes test fail), #2047 (pytest fails at first test with Exit: FATAL), #1777 (sigalrm fails test), and #2479 (Running tests generated by pytest_generate_tests with the -k flag fails.).

[NiklasMM]

I took a quick look and the problem seems to be that ProfileBase._providers is a class attribute and therefore lives at class level. This means it will be created once the class is imported and there is only one dict shared by all instances. (Take a look at this StackOverflow thread on the difference of class and instance attributes)

Since pytest doesn't re-import code for each run, the same class is used in your second test, with _providers already populated. The easiest fix is to move _providers from class to instance level, by moving it to the constructor.

[RobRoseKnows]

🤦‍♂️ Thanks that fixed it.