Problem with running tests in package's `__init__`

[pstiasny]

This might be related to #3749 and/or #3773.

Given the following directory structure:

tests/
  __init__.py (1)
  test_foo.py (2)

and following setup.cfg:

[tool:pytest]
python_files = test_*.py __init__.py

Before py.test 3.7:

• py.test with no arguments would collect tests in (1) and (2)
• py.test tests/ would collect tests in (1) and (2)
• py.test tests/__init__.py would collect tests in (1)

Until 3.8.1 collecting from __init__.py would not work due to #3749.

In 3.8.1 collecting from __init__.py works again, but:

• py.test with no arguments would behave as before, collecting all tests correctly
• py.test tests/ would collect tests in (2), omitting (1)
• py.test tests/__init__.py would collect tests in (1), but also in (2)

[pstiasny]

I updated test_collect_init_tests to demonstrate this behavior, see pstiasny@1e57a5d.

[nicoddemus]

Thanks @pstiasny for the report!

[blueyed]

@kchmck
Thanks for addressing this in #4285.

I think #4241 (in features already) might fix this already.

[blueyed]

If so it might have regressed through fedc785#diff-7eb3d34b4de135c7f717e983c8282210R195.

[blueyed]

Created a backport for master, please test it: #4286.

[nicoddemus]

Fixed by #4285