Establish best practices for errors on warnings - of which some get ignored

[RonnyPfannschmidt]

follow-up to pypa/setuptools-scm#305

it i blatantly missed an ignored exception from a resource warning due to green test results while setting an error for all warnings

pytest should either establish a well known best practice for configuring that, or go yellow on those errors that get ignored by the stdlib for a good reason

[nicoddemus]

I believe this is a duplicate of #2908? If so I propose to close this in favor of #2908 as it contains more information.

[RonnyPfannschmidt]

@nicoddemus its unrelated - the problem here is that i did set a global error on all - but the resoruce warning happens in a place where the exception is ignored by the stdlib, so i missed the problem since i expected a hard failure, instead i got an "exception ignored"

[nicoddemus]

I see, thanks. I think we might catch that once #3251 lands, but more investigation would be needed.

[SnarkBoojum]

Hmmm... this isn't about a deprecation warning, is it?

[RonnyPfannschmidt]

@SnarkBoojum no, its about a RessourceWarning triggering in a place where python does a catchall

to catch that pytest would need to track those exceptions as well and return a error exit state

[Lekensteyn]

The error is written to stderr when Python cannot handle an exception (in __del__ or during GC). Not sure how to properly check for it except explicitly capturing and checking stderr with capsys.

Here is a minimal reproducer:

test_resleak.py:

def test_resleak():
    f = open(__file__)

Output of pytest -sv -Werror:

============================= test session starts ==============================
platform darwin -- Python 3.7.3, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /usr/local/opt/python/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/me/tmp/pyex
plugins: xdist-1.28.0, forked-1.0.2
collecting ... collected 1 item

test_resleak.py::test_resleak PASSED

=========================== 1 passed in 0.01 seconds ===========================
Exception ignored in: <_io.FileIO name='/Users/me/tmp/pyex/test_resleak.py' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/Users/me/tmp/pyex/test_resleak.py' mode='r' encoding='UTF-8'>

[nicoddemus]

I think by that point we are already outside pytest.main() so, I don't see what pytest can do.

[nicoddemus]

I'm closing this for now: the exceptions/warnings mentioned here happen after pytest.main() has exited, so I don't see how we can implement this (if somebody does, please feel free to reopen with a suggestion). 👍