Pytest fails to capture doctest traceback?

[jacob414]

Hello,

I'm seing an issue that I can't explain where doctest output doesn't seem to be captured by Pytest. I develop a library where state changes where I can make an exception happen like this:

import sys
from altered import state, forget
with state(sys.modules, shutil=forget):
    import shutil # shutils artificially removed from `os.modules`
#  Traceback output....

I have created a file that prints a correct checking value when run under regular doctests:

[..]$ ~/opt/vmini/bin/python doctest_issue.py
TestResults(failed=0, attempted=3) # Expected output

But when I run one of my documentation files where an almost identical example appears, it will fail to detect the exception and give a failed test message ending with:

$ ~/opt/vmini/bin/py.test docs/examples.rst
Expected:
    Traceback (most recent call last):
        ...
    KeyError: 'shutil'
Got nothing

My environment:

$ uname -a
Darwin goldskip 16.3.0 Darwin Kernel Version 16.3.0: Thu Nov 17 20:23:58 PST 2016; root:xnu-3789.31.2~1/RELEASE_X86_64 x86_6
$ ~/opt/vmini/bin/python --version
Python 2.7.10
$ ~/opt/vmini/bin/pip list
# [...]
pip (9.0.1)
py (1.4.32)
pytest (3.0.5)
setuptools (33.1.1)
wheel (0.30.0a0)

( also replicated the issue on Linux 4.4.0 / Python 2.7.6).

First I suspected that Pytest's output capture could be playing tricks, but the issue is the same even if I invoke with the -s commandline flag.

I could make a quick replication by doing the following (on the Linux system mentioned above):

(Assuming a fairly empty home directory on a system /w virtualenv installed)

~$ git clone https://github.com/Plexical/altered.states.git
#[...]
$ virtualenv venv
#[...]
$ ~/venv/bin/pip install pytest
#[...]
~$ cd altered.states/
[..]:~/altered.states$ ~/venv/bin/python setup.py develop
#[...]
[..]:~/altered.states$ ~/venv/bin/python doctest_issue.py
TestResults(failed=0, attempted=3)
[..]:~/altered.states$ ~/venv/bin/py.test docs/examples.rst
==================================================== test session starts =====================================================
platform linux2 -- Python 2.7.6, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /home/labbo/altered.states, inifile: pytest.ini
collected 1 items

docs/examples.rst F

========================================================== FAILURES ==========================================================
___________________________________________________ [doctest] examples.rst ___________________________________________________
047     foo
048
049 Deny the existance of a module
050 ------------------------------
051
052 *It'd be much better if it would raise `ImportError` here. Maybee later.*
053
054     >>> import sys
055     >>> from altered import state, forget
056     >>> with state(sys.modules, shutil=forget):
Expected:
    Traceback (most recent call last):
        ...
    KeyError: 'shutil'
Got nothing

/home/labbo/altered.states/docs/examples.rst:56: DocTestFailure
================================================== 1 failed in 0.07 seconds ==================================================

[nicoddemus]

Thanks for the report. I did investigate a little, will take a fresher look tomorrow!

[jacob414]

Ops...

My bad, you can close the issue! Apparently, removing a module from sys.modules isn't the way to block access to (unimported) modules (see e.g. http://stackoverflow.com/q/1350466/288672).

Sorry for bothering you, although thanks for the opportunity to explore the correct way of preventing certain imports!

[nicoddemus]

OK, thanks for following up!