Fixture order considering global states

[Sup3rGeo]

Hi,

This section of the documentation automatic grouping of tests by fixtures instances almost solves one need we are having, related to fixtures dealing with global states.

Problem example

To exemplify, let's simplify the example on the documentation, with seemly independent tests, each one with its fixture (modarg with test_1; and otherarg with test_2).

modarg and otherarg change the same global state.

# content of test_module.py
import pytest

@pytest.fixture(scope="module", params=['a', 'b'])
def modarg(request):
    return request.param

@pytest.fixture(scope="module", params=['a', 'b'])
def otherarg(request):
    return request.param

def test_0(otherarg):
    print("  RUN test0 with otherarg %s" % otherarg)

def test_1(modarg):
    print("  RUN test1 with modarg %s" % modarg)

It is not a problem because pytest orders in such a way that all tests using one fixture are executed, then all tests using the another:

test_example2.py::test_0[a] PASSED
test_example2.py::test_0[b] PASSED
test_example2.py::test_1[a] PASSED
test_example2.py::test_1[b] PASSED

Now, let's say I want to refactor the ['a', 'b'] parameters into a common fixture.
The problem now is that pytest tries to first use the new, highest level uppermost fixture first, now interleaving the execution of the fixtures modarg and otherarg (which is a problem because they are module scoped and are dealing with the same shared resource).

# content of test_module.py
import pytest

@pytest.fixture(scope="module", params=['a', 'b'])
def uppermost(request):
    return request.param

@pytest.fixture(scope="module")
def modarg(uppermost):
    return uppermost

@pytest.fixture(scope="module")
def otherarg(uppermost):
    return uppermost

def test_0(otherarg):
    print("  RUN test0 with otddherarg %s" % otherarg)

def test_1(modarg):
    print("  RUN test1 with modarg %s" % modarg)

test_example.py::test_0[a] PASSED
test_example.py::test_1[a] PASSED
test_example.py::test_0[b] PASSED
test_example.py::test_1[b] PASSED

Needed behaviour

In our situation, the resource is a piece of hardware, and the module-scoped fixture is an expensive set-up of that hardware (start-up and configuration).

For this case, it would solve to be able to specify that a given fixture share a resource - so, when ordering, pytest can make sure that there is no interleaving between the execution of tests that share that same resource.

So in our example, we would possibly have this:

@pytest.fixture(scope="module", shared_resource="myhardware")
def modarg(uppermost):
    return uppermost

@pytest.fixture(scope="module", shared_resource="myhardware")
def otherarg(uppermost):
    return uppermost

And that's it. I would like to hear if this is sensible to add or not, probably more experienced developers might spot any detrimental side effects of this. I would be willing to implement this myself once we decide it could and should be done.

Thanks!

[smarie]

This very interesting request seems quite interdependent with the new order optimization algorithm (#3551), your own param-priority (#3393) plugin, etc.

I am personally interested because in pytest-cases when fixture unions are used, a new parameter 'NOT_USED" is added to fixtures that are not used in some callspecs, so that they are not setup/torn down. So I would like to be sure that there is no particular gotcha around higher-level scope ordering concerning them.

[smarie]

Having a second look at this topic, I think that part of it can be solved by your plugin.

Indeed the main problem comes from the fact that pytest tries to minimize the number of setup/teardown for fixture uppermost, while what you really want is to reduce the number of setup/teardown for fixtures otherarg / modarg.

I tried this:

from pytest_param_priority import parameter_priority

# content of test_module.py
import pytest

@parameter_priority(1)
@pytest.fixture(scope="module", params=['a', 'b'])
def uppermost(request):
    return request.param

@parameter_priority(0)
@pytest.fixture(scope="module")
def modarg(uppermost):
    return uppermost

@parameter_priority(0)
@pytest.fixture(scope="module")
def otherarg(uppermost):
    return uppermost

def test_0(otherarg):
    print("  RUN test0 with otddherarg %s" % otherarg)

def test_1(modarg):
    print("  RUN test1 with modarg %s" % modarg)

but... it does not work :(

Of course this would not solve the "shared resource" issue but at least it would solve the "modarg and otherarg take much more time to setup/teardown" issue

[Sup3rGeo]

I tried this:

This won't have any effect in the pytest-param-priority plugin because what it is looking for are not fixtures per se, but their parameters instead. And this is because when ordering, pytest only deals with test functions themselves and its parameters (which might be inherited from the fixtures it depends on).

That's why this will not have any effect:

@parameter_priority(0)
@pytest.fixture(scope="module")
def modarg(uppermost):
    return uppermost

Because there is are parameter marks in this fixture function.

[smarie]

ah i see, thanks for clarifying this!