Automatic grouping of tests by fixture instances broken when using metafunc.parametrize

[pytestbot]

Originally reported by: BitBucket: davidkr, GitHub: davidkr

---

from
Automatic grouping of tests by fixture instances

pytest minimizes the number of active fixtures during test runs. If you have a parametrized
fixture, then all the tests using it will first execute with one instance and then finalizers are
called before the next fixture instance is created. Among other things, this eases testing of
applications which create and use global state.

However the grouping doesn't seem to be working when using metafunc.parametrize. Below is an example of what I mean. Here's the test code...

#!python
import pytest

@pytest.yield_fixture(scope="session",
                      params=["big mac", "whopper"])
def burger(request):
    yield request.param

@pytest.yield_fixture(scope="function",
                      params=["curlyfries", "regularfries"])
def fries(request):
    yield request.param

def test_burgers(burger):
    print "test_burgers: {0}".format(burger)

def test_combo1(burger, fries):
    print("test_combo1: {0}{1}".format(burger, fries))

The output from running the tests..

test_grouping.py::test_burgers[big mac] PASSED
test_grouping.py::test_combo1[big mac-curlyfries] PASSED
test_grouping.py::test_combo1[big mac-regularfries] PASSED
test_grouping.py::test_burgers[whopper] PASSED
test_grouping.py::test_combo1[whopper-curlyfries] PASSED
test_grouping.py::test_combo1[whopper-regularfries] PASSED

So the things get group as I would expect. However I use metafunc to parametrize the burger fixture instead of the decorator by doing this...

#!python
def pytest_generate_tests(metafunc):
    if "burger" in metafunc.fixturenames:
        metafunc.parametrize("burger", argvalues=["big mac", "whopper"])

..then the order changes and all instances of test_burger get run first and the output looks like this....

test_grouping.py::test_burgers[big mac] PASSED
test_grouping.py::test_burgers[whopper] PASSED
test_grouping.py::test_combo1[big mac-curlyfries] PASSED
test_grouping.py::test_combo1[big mac-regularfries] PASSED
test_grouping.py::test_combo1[whopper-curlyfries] PASSED
test_grouping.py::test_combo1[whopper-regularfries] PASSED

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/661

[kvas-it]

I was looking into this today. metafunc.parametrize takes a parameter called scope and if you set it to "session", the grouping of the tests will be the same as with parametrization via decorator parameters (the first example).
However, the default behaviour doesn't seem to be sane. Tomorrow, I'll see if we can make it do the right thing by default.

[nicoddemus]

Sounds good @kvas-it, thanks for looking into this man! Let me know if I can help out, although tomorrow I'm not sure if I will available much.

[kvas-it]

I did a bunch of experiments with this today. Here's a brief report of key findings. TLDR: The behaviour described here is correct because second example is parametrizing the functions and not the fixture.

However, we might want to take care of some unobvious sharp corners of the parametrization API:

• Direct parametrization of test functions in anything but function scope doesn't make sense to me. Perhaps we should show a warning if it's attempted.
• For indirect parametrization the scope should match the scope of one of the existing fixtures. I guess we could issue a warning if it doesn't. Autodetecting the scope sounds like a nice fix for this, but I don't think we can do it for two reasons: (1) it will change behaviour of existing test suites and (2) if we parametrize a non-function scoped fixture giving it different parameters for different tests, things will behave confusingly.
• Maybe we can improve the documentation of parametrization so that it's more clear from it what would happen in the second example from this issue.

[The-Compiler]

@kvas-it Do you think anything of this should block 3.0, or should we change the milestone for this to 4.0?

[kvas-it]

IMO this shouldn't block anything because I think we don't want to change current behaviour and would only add warnings for some confusing cases + update the documentation. So it could be in any minor release later too.
I was actually waiting for someone to confirm my plan above, so, @The-Compiler, if you feel qualified, tell me what you think.

[The-Compiler]

I don't really do, sorry - I haven't used parametrization other than @pytest.mark.parametrize and direct straightforward fixture parametrization.