[8.1.x][8.3.x] testpaths not taken into account for initial conftests loading when set through configuration file?

[vmonteco]

I'm trying to set up a custom command line option with the pytest_addoption hook in a non-root conftest.py file but it looks like that somehow my conftest isn't taken into account as an initial one if the testpaths is set through a configuration file (pyproject.toml in my example but it was also reproduced with a pytest.ini).
However, things work if the testpaths is explicitly set by command line argument.

Versions used are:

• Python 3.10.1
• Pytest 8.1.1 (also reproduced with 8.3.2)

$ pip freeze
exceptiongroup==1.2.2
iniconfig==2.0.0
packaging==24.1
pluggy==1.5.0
pytest==8.1.1
tomli==2.0.1
$ python --version
Python 3.10.13
$ 

The OS is a Linux environment:

$ uname -a
Linux vmonteco-desktop 6.8.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 16 Mar 2024 17:15:35 +0000 x86_64 GNU/Linux
$ 

I made a SO post on this issue yesterday and someone pointed out a similar issue (#10988). Could it be a regression?

MRE :

Command pytest --foo bar run in a directory containing the files:

• ./pyproject.toml:

  [tool.pytest.ini_options]
  testpaths = [
      "my_project/my_tests",
  ]

• ./my_project/my_tests/conftest.py:

  import pytest
  
  def pytest_addoption(parser):
      parser.addoption("--foo", action="store")
  
  @pytest.fixture
  def my_val(request):
      return request.config.getoption("--foo")

• ./my_project/my_tests/test_foo.py:

  def test_foo(my_val):
      assert my_val == "foo"

• An empty ./my_project/my_tests/__init__.py file.

Expected behaviour:

An output similar to the one a simple workaround (giving the testpath explicitly by command line) gives:

$ pytest --foo bar my_project/my_tests
=============================== test session starts ===============================
platform linux -- Python 3.10.13, pytest-8.1.1, pluggy-1.5.0
rootdir: /home/vmonteco/code/MREs/pytest__addoption__pyproject_toml/01_adding_pyproject_toml
configfile: pyproject.toml
collected 1 item                                                                  

my_project/my_tests/test_foo.py F                                           [100%]

==================================== FAILURES =====================================
____________________________________ test_foo _____________________________________

my_val = 'bar'

    def test_foo(my_val):
>       assert my_val == "foo"
E       AssertionError: assert 'bar' == 'foo'
E         
E         - foo
E         + bar

my_project/my_tests/test_foo.py:2: AssertionError
============================= short test summary info =============================
FAILED my_project/my_tests/test_foo.py::test_foo - AssertionError: assert 'bar' == 'foo'
================================ 1 failed in 0.02s ================================
$

Observed behaviour:

I get the following usage error:

$ pytest --foo bar                  
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --foo
  inifile: /home/vmonteco/code/MREs/pytest__addoption__pyproject_toml/02_attempt_to_solve/pyproject.toml
  rootdir: /home/vmonteco/code/MREs/pytest__addoption__pyproject_toml/02_attempt_to_solve

$ 

[vmonteco]

I'm currently trying to take care of it and I've set up a dev environment but I'm not able to produce a test that could fail on this.

There already is a test covering similar cases but in doubt I've tried the following that is very similar (if not duplicate) to test_collection.py::test_initial_conftests_with_testpaths.~~

But it passes.

def test_initial_conftests_with_testpaths_bis(pytester: Pytester) -> None:
    """
    Trying to reproduce #12696.
    """
    # Create the pyproject.toml.
    pytester.path.joinpath("pyproject.toml").write_text(
        textwrap.dedent(
            """
            [tool.pytest.ini_options]
            testpaths = [
                "my_project/my_tests",
            ]
            """
        ),
        encoding="utf-8",
    )
    
    # Create the conftest.
    pytester.mkdir("my_project")
    p = pytester.mkdir("my_project/my_tests")
    p.joinpath("conftest.py").write_text(
        textwrap.dedent(
            """
            def pytest_sessionstart(session):
                raise Exception("pytest_sessionstart hook successfully run")
            """
        ),
        encoding="utf-8",
    )

    result = pytester.runpytest()
    assert result.ret == ExitCode.INTERNAL_ERROR
    result.stdout.fnmatch_lines(
        "INTERNALERROR* Exception: pytest_sessionstart hook successfully run"
    )

    # No fallback.
    result = pytester.runpytest(".")
    assert result.ret == ExitCode.NO_TESTS_COLLECTED

So I'm looking for hints or alternate hypotheses to explain the behavior I observe.

---

Erratum:

I think I didn't go deep enough.
WIP

[vmonteco]

I tried to make and run the following tests to reproduce the usage error but those tests pass.

class TestAddoptionHookInNonRootConftest:
    """
    Trying to reproduce #12696.
    """

    def test_pytest_addoption_hook_with_command_line_testpaths(self, pytester: Pytester) -> None:
        # Create the conftest.
        pytester.mkdir("my_project")
        p = pytester.mkdir("my_project/my_tests")
        
        p.joinpath("conftest.py").write_text(
            textwrap.dedent(
                f"""
                def pytest_addoption(parser):
                    parser.addoption("--XX", action="store_true")
                """
            ),
            encoding="utf-8",
        )
        
        # No command line args - falls back to testpaths.
        result = pytester.runpytest("my_project/my_tests", "--XX")
        assert result.ret == ExitCode.NO_TESTS_COLLECTED

    def test_pytest_addoption_hook_with_configured_testpaths(self, pytester: Pytester) -> None:
        # Creating the pyproject.toml file.
        pytester.path.joinpath("pyproject.toml").write_text(
            textwrap.dedent(
                """
                [tool.pytest.ini_options]
                testpaths = [
                    "my_project/my_tests",
                ]
                """
            ),
            encoding="utf-8",
        )
        
        # Create the conftest.
        pytester.mkdir("my_project")
        p = pytester.mkdir("my_project/my_tests")
        
        p.joinpath("conftest.py").write_text(
            textwrap.dedent(
                f"""
                def pytest_addoption(parser):
                    parser.addoption("--XX", action="store_true")
                """
            ),
            encoding="utf-8",
        )
        
        # No command line args - should fall back to configured testpaths.
        result = pytester.runpytest("--XX")
        assert result.ret == ExitCode.NO_TESTS_COLLECTED

Perhaps it's not a bug finally and rather something I messed up. Perhaps I should fill a support request instead?

[vmonteco]

Oh. * Facepalm * I just found out I was indeed a usage error.

pytest --foo bar is a bad usage.
pytest --foo=bar is the good one.

My bad and sorry for the inconvenience!

[wimglenn]

Related: #8846