Generate tests parametrizing many cases

[jbrockmendel]

Over in dateutil we've been slowly transitioning towards using pytest. In my first attempt to use pytest.mark.parametrize I made some absurd number of combinations by specifying a half dozen parameters and iterating over large ranges of possible dates. Millions of tests, RAM use exploded during test collection, and I had to kill the process. Painless learning experience as these thing go.

Is there a supported way to build these tests generator-style instead of list-style? Something analogous to nose's yield-base tests?

[pytestbot]

GitMate.io thinks the contributor most likely able to help you is @RonnyPfannschmidt.

[RonnyPfannschmidt]

for generating millions of examples and verifying those, its best to use hypothesis and its strategies,
pytest would have to collect all those tests with all the metadata

collection happens always before test running

[nicoddemus]

I think the short answer for now is that pytest currently doesn't support generator-style tests because the current model requires that collection happens before the run phase. I don't see this changing because it is a major shift in design and would probably signify major breakage.

I suppose an alternative would be to support subtest-style tests in pytest (#1367), because you would have a single test item and multiple reports. This is probably something possible to accomplish because currently for each test item we already have a multiple reports, and some plugins (pytest-repeat for example) take this further and generate multiple "call" reports for each item anyway.

I'm closing this for now but feel free to follow up with any questions you may have.