pytest tests explicitly check for ImportError, fails on Python 3.6

[hroncok]

I'm trying to run the test suite on Python 3.6.

Here https://github.com/pytest-dev/pytest/blob/3.0.5/testing/test_doctest.py#L215 the tests try to check if the text ImportError is in the output, but on Python 3.6, I'm getting ModuleNotFoundError instead, which I think is OK and the test should work with ModuleNotFoundError as well.

 ERROR at setup of TestTrialUnittest.test_trial_testcase_runtest_not_collected _

cls = <class 'test_unittest.TestTrialUnittest'>

    def setup_class(cls):
>       cls.ut = pytest.importorskip("twisted.trial.unittest")

testing/test_unittest.py:390: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.6/site-packages/twisted/trial/unittest.py:12: in <module>
    from twisted.trial._synctest import (
/usr/lib64/python3.6/site-packages/twisted/trial/_synctest.py:23: in <module>
    from twisted.trial import itrial, util
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    
    from __future__ import division, absolute_import
    
>   import zope.interface as zi
E   AttributeError: module 'zope' has no attribute 'interface'

/usr/lib64/python3.6/site-packages/twisted/trial/itrial.py:12: AttributeError
_____ ERROR at setup of TestTrialUnittest.test_trial_exceptions_with_skips _____

cls = <class 'test_unittest.TestTrialUnittest'>

    def setup_class(cls):
>       cls.ut = pytest.importorskip("twisted.trial.unittest")

testing/test_unittest.py:390: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.6/site-packages/twisted/trial/unittest.py:12: in <module>
    from twisted.trial._synctest import (
/usr/lib64/python3.6/site-packages/twisted/trial/_synctest.py:23: in <module>
    from twisted.trial import itrial, util
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    
    from __future__ import division, absolute_import
    
>   import zope.interface as zi
E   AttributeError: module 'zope' has no attribute 'interface'

/usr/lib64/python3.6/site-packages/twisted/trial/itrial.py:12: AttributeError
_____________ ERROR at setup of TestTrialUnittest.test_trial_error _____________

cls = <class 'test_unittest.TestTrialUnittest'>

    def setup_class(cls):
>       cls.ut = pytest.importorskip("twisted.trial.unittest")

testing/test_unittest.py:390: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.6/site-packages/twisted/trial/unittest.py:12: in <module>
    from twisted.trial._synctest import (
/usr/lib64/python3.6/site-packages/twisted/trial/_synctest.py:23: in <module>
    from twisted.trial import itrial, util
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    
    from __future__ import division, absolute_import
    
>   import zope.interface as zi
E   AttributeError: module 'zope' has no attribute 'interface'

/usr/lib64/python3.6/site-packages/twisted/trial/itrial.py:12: AttributeError
______________ ERROR at setup of TestTrialUnittest.test_trial_pdb ______________

cls = <class 'test_unittest.TestTrialUnittest'>

    def setup_class(cls):
>       cls.ut = pytest.importorskip("twisted.trial.unittest")

testing/test_unittest.py:390: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.6/site-packages/twisted/trial/unittest.py:12: in <module>
    from twisted.trial._synctest import (
/usr/lib64/python3.6/site-packages/twisted/trial/_synctest.py:23: in <module>
    from twisted.trial import itrial, util
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    
    from __future__ import division, absolute_import
    
>   import zope.interface as zi
E   AttributeError: module 'zope' has no attribute 'interface'

/usr/lib64/python3.6/site-packages/twisted/trial/itrial.py:12: AttributeError
=================================== FAILURES ===================================
_____________ TestDoctests.test_doctest_unex_importerror_only_txt ______________

self = <test_doctest.TestDoctests object at 0x7ff4f5df93c8>
testdir = <Testdir local('/tmp/pytest-of-mockbuild/pytest-15/testdir/test_doctest_unex_importerror_only_txt0')>

    def test_doctest_unex_importerror_only_txt(self, testdir):
        testdir.maketxtfile("""
                >>> import asdalsdkjaslkdjasd
                >>>
            """)
        result = testdir.runpytest()
        # doctest is never executed because of error during hello.py collection
        result.stdout.fnmatch_lines([
            "*>>> import asdals*",
            "*UNEXPECTED*ImportError*",
>           "ImportError: No module named *asdal*",
        ])
E       Failed: nomatch: '*>>> import asdals*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.6.0rc1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0'
E           and: 'rootdir: /tmp/pytest-of-mockbuild/pytest-15/testdir/test_doctest_unex_importerror_only_txt0, inifile: '
E           and: 'plugins: hypothesis-3.4.0'
E           and: 'collected 1 items'
E           and: ''
E           and: 'test_doctest_unex_importerror_only_txt.txt F'
E           and: ''
E           and: '=================================== FAILURES ==================================='
E           and: '_____________ [doctest] test_doctest_unex_importerror_only_txt.txt _____________'
E       fnmatch: '*>>> import asdals*'
E          with: '001 >>> import asdalsdkjaslkdjasd'
E       nomatch: '*UNEXPECTED*ImportError*'
E           and: 'UNEXPECTED EXCEPTION: ModuleNotFoundError("No module named \'asdalsdkjaslkdjasd\'",)'
E           and: 'Traceback (most recent call last):'
E           and: ''
E           and: '  File "/usr/lib64/python3.6/doctest.py", line 1330, in __run'
E           and: '    compileflags, 1), test.globs)'
E           and: ''
E           and: '  File "<doctest test_doctest_unex_importerror_only_txt.txt[0]>", line 1, in <module>'
E           and: ''
E           and: "ModuleNotFoundError: No module named 'asdalsdkjaslkdjasd'"
E           and: ''
E           and: '/tmp/pytest-of-mockbuild/pytest-15/testdir/test_doctest_unex_importerror_only_txt0/test_doctest_unex_importerror_only_txt.txt:1: UnexpectedException'
E           and: '=========================== 1 failed in 0.01 seconds ==========================='
E           and: ''
E       remains unmatched: '*UNEXPECTED*ImportError*'

/builddir/build/BUILD/pytest-3.0.5/python3/testing/test_doctest.py:215: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.6.0rc1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /tmp/pytest-of-mockbuild/pytest-15/testdir/test_doctest_unex_importerror_only_txt0, inifile: 
plugins: hypothesis-3.4.0
collected 1 items

test_doctest_unex_importerror_only_txt.txt F

=================================== FAILURES ===================================
_____________ [doctest] test_doctest_unex_importerror_only_txt.txt _____________
001 >>> import asdalsdkjaslkdjasd
UNEXPECTED EXCEPTION: ModuleNotFoundError("No module named 'asdalsdkjaslkdjasd'",)
Traceback (most recent call last):

  File "/usr/lib64/python3.6/doctest.py", line 1330, in __run
    compileflags, 1), test.globs)

  File "<doctest test_doctest_unex_importerror_only_txt.txt[0]>", line 1, in <module>

ModuleNotFoundError: No module named 'asdalsdkjaslkdjasd'

/tmp/pytest-of-mockbuild/pytest-15/testdir/test_doctest_unex_importerror_only_txt0/test_doctest_unex_importerror_only_txt.txt:1: UnexpectedException
=========================== 1 failed in 0.01 seconds ===========================
____________ TestDoctests.test_doctest_unex_importerror_with_module ____________

self = <test_doctest.TestDoctests object at 0x7ff4f6043be0>
testdir = <Testdir local('/tmp/pytest-of-mockbuild/pytest-15/testdir/test_doctest_unex_importerror_with_module0')>

    def test_doctest_unex_importerror_with_module(self, testdir):
        testdir.tmpdir.join("hello.py").write(_pytest._code.Source("""
                import asdalsdkjaslkdjasd
            """))
        testdir.maketxtfile("""
                >>> import hello
                >>>
            """)
        result = testdir.runpytest("--doctest-modules")
        # doctest is never executed because of error during hello.py collection
        result.stdout.fnmatch_lines([
            "*ERROR collecting hello.py*",
            "*ImportError: No module named *asdals*",
>           "*Interrupted: 1 errors during collection*",
        ])
E       Failed: nomatch: '*ERROR collecting hello.py*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.6.0rc1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0'
E           and: 'rootdir: /tmp/pytest-of-mockbuild/pytest-15/testdir/test_doctest_unex_importerror_with_module0, inifile: '
E           and: 'plugins: hypothesis-3.4.0'
E           and: 'collected 1 items / 1 errors'
E           and: ''
E           and: '==================================== ERRORS ===================================='
E       fnmatch: '*ERROR collecting hello.py*'
E          with: '__________________________ ERROR collecting hello.py ___________________________'
E       nomatch: '*ImportError: No module named *asdals*'
E           and: 'hello.py:2: in <module>'
E           and: '    import asdalsdkjaslkdjasd'
E           and: "E   ModuleNotFoundError: No module named 'asdalsdkjaslkdjasd'"
E           and: '!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!'
E           and: '=========================== 1 error in 0.13 seconds ============================'
E           and: ''
E       remains unmatched: '*ImportError: No module named *asdals*'

/builddir/build/BUILD/pytest-3.0.5/python3/testing/test_doctest.py:231: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.6.0rc1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /tmp/pytest-of-mockbuild/pytest-15/testdir/test_doctest_unex_importerror_with_module0, inifile: 
plugins: hypothesis-3.4.0
collected 1 items / 1 errors

==================================== ERRORS ====================================
__________________________ ERROR collecting hello.py ___________________________
hello.py:2: in <module>
    import asdalsdkjaslkdjasd
E   ModuleNotFoundError: No module named 'asdalsdkjaslkdjasd'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.13 seconds ============================
____________________ test_trial_testfunction_skip_property _____________________

testdir = <Testdir local('/tmp/pytest-of-mockbuild/pytest-15/testdir/test_trial_testfunction_skip_property0')>

    def test_trial_testfunction_skip_property(testdir):
>       pytest.importorskip('twisted.trial.unittest')

/builddir/build/BUILD/pytest-3.0.5/python3/testing/test_unittest.py:350: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.6/site-packages/twisted/trial/unittest.py:12: in <module>
    from twisted.trial._synctest import (
/usr/lib64/python3.6/site-packages/twisted/trial/_synctest.py:23: in <module>
    from twisted.trial import itrial, util
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    
    from __future__ import division, absolute_import
    
>   import zope.interface as zi
E   AttributeError: module 'zope' has no attribute 'interface'

/usr/lib64/python3.6/site-packages/twisted/trial/itrial.py:12: AttributeError
______________________ test_trial_testcase_todo_property _______________________

testdir = <Testdir local('/tmp/pytest-of-mockbuild/pytest-15/testdir/test_trial_testcase_todo_property0')>

    def test_trial_testcase_todo_property(testdir):
>       pytest.importorskip('twisted.trial.unittest')

/builddir/build/BUILD/pytest-3.0.5/python3/testing/test_unittest.py:363: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.6/site-packages/twisted/trial/unittest.py:12: in <module>
    from twisted.trial._synctest import (
/usr/lib64/python3.6/site-packages/twisted/trial/_synctest.py:23: in <module>
    from twisted.trial import itrial, util
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    
    from __future__ import division, absolute_import
    
>   import zope.interface as zi
E   AttributeError: module 'zope' has no attribute 'interface'

/usr/lib64/python3.6/site-packages/twisted/trial/itrial.py:12: AttributeError
____________________ test_trial_testfunction_todo_property _____________________

testdir = <Testdir local('/tmp/pytest-of-mockbuild/pytest-15/testdir/test_trial_testfunction_todo_property0')>

    def test_trial_testfunction_todo_property(testdir):
>       pytest.importorskip('twisted.trial.unittest')

/builddir/build/BUILD/pytest-3.0.5/python3/testing/test_unittest.py:376: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.6/site-packages/twisted/trial/unittest.py:12: in <module>
    from twisted.trial._synctest import (
/usr/lib64/python3.6/site-packages/twisted/trial/_synctest.py:23: in <module>
    from twisted.trial import itrial, util
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    
    from __future__ import division, absolute_import
    
>   import zope.interface as zi
E   AttributeError: module 'zope' has no attribute 'interface'

/usr/lib64/python3.6/site-packages/twisted/trial/itrial.py:12: AttributeError
=== 5 failed, 1668 passed, 44 skipped, 13 xfailed, 4 error in 80.77 seconds ====

Note that the other issue with zope interaces is weird as well because I'm confident I have those, but I will explore that later. Might be related to #1989

Unfortunately from this environment, I cannot call pip freeze. It's Fedora rawhide (26) and Python 3.6.0rc1.

[hroncok]

This fixes the thing (except the zope part), but makes the tests less specific, so I'm unsure, if it is a right approach.

diff --git a/testing/test_doctest.py b/testing/test_doctest.py
index 4ea2cc5..a82071c 100644
--- a/testing/test_doctest.py
+++ b/testing/test_doctest.py
@@ -211,8 +211,8 @@ class TestDoctests:
         # doctest is never executed because of error during hello.py collection
         result.stdout.fnmatch_lines([
             "*>>> import asdals*",
-            "*UNEXPECTED*ImportError*",
-            "ImportError: No module named *asdal*",
+            "*UNEXPECTED*Error*",
+            "*Error: No module named *asdal*",
         ])
 
     def test_doctest_unex_importerror_with_module(self, testdir):
@@ -227,7 +227,7 @@ class TestDoctests:
         # doctest is never executed because of error during hello.py collection
         result.stdout.fnmatch_lines([
             "*ERROR collecting hello.py*",
-            "*ImportError: No module named *asdals*",
+            "*Error: No module named *asdals*",
             "*Interrupted: 1 errors during collection*",
         ])

I was also able to do pip freeze:

argcomplete==1.1.1
attrs==16.1.0
cffi==1.8.3
cryptography==1.5.3
decorator==4.0.10
funcsigs==1.0.2
hypothesis==3.4.0
idna==2.1
Jinja2==2.8
MarkupSafe==0.23
mock==2.0.0
nose==1.3.7
pbr==1.10.0
ply==3.9
py==1.4.31
pyasn1==0.1.9
pyasn1-modules==0.0.8
pycparser==2.14
pycrypto==2.6.1
pyOpenSSL==16.2.0
pyserial==3.1.1
service-identity==16.0.0
six==1.10.0
Twisted==16.4.1
zope.event==4.2.0
zope.interface==4.3.2

[nicoddemus]

Thanks for reporting this!

This reminds me that we should start testing pytest automatically on Python 3.6 as well, created an issue to track this: #2134.

Your patch looks fine to me FWIW.