Adding INI file option testpaths through addinivalue_line does not give expected behavior.

[Drew-Ack]

Im trying to allow users to select specific test directories trough the command line that pytest will run.

List of steps in my program

1. Parse command line input
2. Use the parser.addini function inside pytest_addoption hook to add testpaths ini option

@pytest.hookimpl()
def pytest_addoption(parser):
    # Allows the user to select the test suite they want to run, or all test suites as default.
    parser.addoption("--suite", action="store", default="all"
                     , choices=['all', 'flow', 'legacy']
                     , help="Choose which test suite to run.")

    parser.addini("testpaths", 'help', type="pathlist")

3. Inside pytest_configure hook use config.addinivalue_line to add the parsed command line option to the ini value created in step 2.

@pytest.hookimpl()
def pytest_configure(config):
    print("Determining test directory")
    suite = config.getoption("--suite")

    try:
        if suite == "flow":
            config.addinivalue_line("testpaths", config.rootdir+"tests/flow_test_cases")

        elif suite == "legacy":
            config.addinivalue_line("testpaths", "tests/legacy_test_cases")

        elif suite == "all":
            config.addinivalue_line("testpaths", "tests/flow_test_cases")
            config.addinivalue_line("testpaths", "tests/legacy_test_cases")

        print("Getting testpaths ini option")
        print(config.getini("testpaths"))

    except Exception as E:
        print(E.args)

4. pytest should look at the ini value testpaths to select specific directories to collect test from.

I succeed in step 1 and 2 and most likely 3, but I cannot get step 4 to succeed. Pytest just collects all tests instead of specific directories.

platform darwin -- Python 3.6.2, pytest-3.3.2, py-1.4.34, pluggy-0.4.0

[pytestbot]

GitMate.io thinks the contributors most likely able to help are @nicoddemus, and @RonnyPfannschmidt.

[nicoddemus]

Hi @Drew-Ack sorry for the delay.

It's not possible to change testpaths during pytest_configure because by the time pytest_configure has been called the testpaths options has already been used by pytest's collection mechanism.

If you want an alternative, I suggest using the k option instead to filter based on the tests names.

pytest -k flow_test_cases

Another approach would be to implement a pytest_collection_modify_items hook which sets custom markers on items based on their directories so users can type this:

pytest -m flow

[nicoddemus]

Closing as this has not seen activity in awhile.