Trying to ignore custom warning from the package to test raises errors

[Bibo-Joshi]

Hi!

I'm trying to upgrade my test suite from v4.2.0 to the current version (6.2.2) and facing issues regarding filterwarnings. Specifically I'm trying to ignore the custom deprecation warning defined in the package that I'm testing. Specifying that in setup.cfg as

[tool:pytest]
filterwarnings =
    ignore::package.CustomWarning

results in a traceback ending on

...
File "…/lib/python3.8/warnings.py", line 262, in _getcategory
    raise _OptionError("invalid module name: %r" % (module,)) from None
warnings._OptionError: invalid module name: 'package'

i.e. the std warnings library can't find the package. The full traceback is here. Also it doesn't seem to matter if the configuration is done via setup.cfg or pyproject.toml (didn't try pytest.ini …)

I'm not sure if I can change the configuration in some way. However, this still works in v6.1.x and IISC the failure in v6.2.x is due to #7700. That's why I chose the bug template rather than the question template …

To clarify: Calling warnings._getcategory('package.CustomWarnings') manually in the same directory I'm running pytest from secceeds. So I guess it's some sort of cwd problem.

For now the workaround suggested at #7525 (comment) seems to work, i.e.

def pytest_runtestloop(session):
    session.add_marker(pytest.mark.filterwarnings('ignore::package.CustomWarning'))

does the trick. Still, I'd be happy if I could specify this in the config file again :)

Output of pip list

Package      Version
------------ -------
atomicwrites 1.4.0
attrs        20.3.0
colorama     0.4.4
iniconfig    1.1.1
packaging    20.9
pip          21.0.1
pluggy       0.13.1
py           1.10.0
pyparsing    2.4.7
pytest       6.2.2
setuptools   53.0.0

pytest & OS:

v6.2.2 on Windows 10

MWE

Can be found at https://github.com/Bibo-Joshi/ci-test

[bluetech]

Thanks for a great bug report @Bibo-Joshi.

I think the issue this:

1. pytest sets up filterwarnings when it starts up (ignore::package.CustomWarning in this case)
2. filterwarnings imports the warning's module (package)
3. The import is done with __import__ directly, which uses the normal python sys.path to look up the module (-> doesn't use the special code pytest uses to import tests).
4. When running pytest ..., the CWD is not added to sys.path, hence package is not found.

To verify this, I used python -m pytest ..., which does add CWD to the sys.path, and this does work.

I am not sure what is the best behavior here, I need to think about it. I wonder what others think.

[Bibo-Joshi]

Hey, thanks for the quick reply :) I can confirm that python -m pytest … works. I'm certainly not familiar with the internals of pytest and can'd judge how complicated it would be to remove the need for python -m or which other side effects that would have. I'm rather sure though that I would constantly forget to prepend it 😬

[nicoddemus]

However, this still works in v6.1.x and IISC the failure in v6.2.x is due to #7700.

You are correct, previously in 6.1.x that warning was not captured, but it is now.

I can confirm that python -m pytest … works.

To reiterate, python -m pytest adds CWD to sys.path, while pytest ... doesn't. This is not particular to pytest, but every python command line application works the same way.

I suggest to actually install the module you are testing in the virtual environment using pip install -e., this will make package importable independently of where you are executing python or pytest. This is considered the good practice nowadays. You can read more about this in: https://docs.pytest.org/en/stable/goodpractices.html?highlight=editable#install-package-with-pip

HTH 👍