New test failures with Python 3.9.0a6 #7161

hroncok · 2020-05-04T18:42:09Z

After upgrading to Python 3.9.0a6, I see 2 new test failures, both in testing/code/test_source.py.

This is Fedora 32, CPython 3.9.0a6, with tox -e py39, master branch @ 80e5098.

$ .tox/py39/bin/pip list
Package          Version                
---------------- -----------------------
argcomplete      1.11.1                 
attrs            19.3.0                 
certifi          2020.4.5.1             
chardet          3.0.4                  
elementpath      1.4.4                  
hypothesis       5.10.5                 
idna             2.9                    
mock             4.0.2                  
more-itertools   8.2.0                  
nose             1.3.7                  
packaging        20.3                   
pip              19.3.1                 
pluggy           0.13.1                 
py               1.8.1                  
Pygments         2.6.1                  
pyparsing        2.4.7                  
pytest           5.4.1.dev171+g80e509840
requests         2.23.0                 
setuptools       41.6.0                 
six              1.14.0                 
sortedcontainers 2.1.0                  
urllib3          1.25.9                 
wcwidth          0.1.9                  
wheel            0.33.6                 
xmlschema        1.1.3

a detailed description of the bug or suggestion
output of pip list from the virtual environment you are using
pytest and operating system versions
minimal example if possible

$ tox -e py39 -- -k "test_syntaxerror_rerepresentation or test_getfslineno"
py39 inst-nodeps: .../pytest/.tox/.tmp/package/1/pytest-5.4.1.dev171+g80e509840.tar.gz
py39 installed: argcomplete==1.11.1,attrs==19.3.0,certifi==2020.4.5.1,chardet==3.0.4,elementpath==1.4.4,hypothesis==5.10.5,idna==2.9,mock==4.0.2,more-itertools==8.2.0,nose==1.3.7,packaging==20.3,pluggy==0.13.1,py==1.8.1,Pygments==2.6.1,pyparsing==2.4.7,pytest==5.0.2.dev41+g183750fa8,requests==2.23.0,six==1.14.0,sortedcontainers==2.1.0,urllib3==1.25.9,wcwidth==0.1.9,xmlschema==1.1.3
py39 run-test-pre: PYTHONHASHSEED='3233708815'
py39 run-test: commands[0] | pytest -k 'test_syntaxerror_rerepresentation or test_getfslineno'
============================= test session starts ==============================
platform linux -- Python 3.9.0a6, pytest-5.4.1.dev171+g80e509840, py-1.8.1, pluggy-0.13.1
cachedir: .tox/py39/.pytest_cache
rootdir: .../pytest, inifile: tox.ini, testpaths: testing
plugins: hypothesis-5.10.5
collected 2688 items / 2686 deselected / 2 selected

testing/code/test_source.py FF                                           [100%]

=================================== FAILURES ===================================
______________________ test_syntaxerror_rerepresentation _______________________

    def test_syntaxerror_rerepresentation() -> None:
        ex = pytest.raises(SyntaxError, _pytest._code.compile, "xyz xyz")
        assert ex is not None
        assert ex.value.lineno == 1
        assert ex.value.offset in {5, 7}  # cpython: 7, pypy3.6 7.1.1: 5
>       assert ex.value.text == "xyz xyz\n"
E       AssertionError: assert 'xyz xyz' == 'xyz xyz\n'
E         - xyz xyz
E         ?        -
E         + xyz xyz

testing/code/test_source.py:125: AssertionError
_______________________________ test_getfslineno _______________________________

    def test_getfslineno() -> None:
        def f(x) -> None:
            raise NotImplementedError()
    
        fspath, lineno = getfslineno(f)
    
        assert isinstance(fspath, py.path.local)
        assert fspath.basename == "test_source.py"
        assert lineno == f.__code__.co_firstlineno - 1  # see findsource
    
        class A:
            pass
    
        fspath, lineno = getfslineno(A)
    
        _, A_lineno = inspect.findsource(A)
        assert isinstance(fspath, py.path.local)
        assert fspath.basename == "test_source.py"
        assert lineno == A_lineno
    
        assert getfslineno(3) == ("", -1)
    
        class B:
            pass
    
        B.__name__ = "B2"
>       assert getfslineno(B)[1] == -1
E       assert 520 == -1

testing/code/test_source.py:525: AssertionError
=========================== short test summary info ============================
FAILED testing/code/test_source.py::test_syntaxerror_rerepresentation - Asser...
FAILED testing/code/test_source.py::test_getfslineno - assert 520 == -1
====================== 2 failed, 2686 deselected in 3.01s ======================
ERROR: InvocationError for command .../pytest/.tox/py39/bin/pytest -k 'test_syntaxerror_rerepresentation or test_getfslineno' (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py39: commands failed

The text was updated successfully, but these errors were encountered:

nicoddemus · 2020-05-04T22:48:59Z

Thanks @hroncok!

We will take a look at this once we can get some version of Python 3.9 running on CI.

hroncok · 2020-05-04T22:56:24Z

I think the first failure might be "fixed" by an rstrip call, however I haven't really grasped the second one :(

asottile · 2020-05-06T19:26:58Z

@nicoddemus we can probably set up either 3.9-tagged (ppa:deadsnakes/ppa) or 3.9-nightly (ppa:deadsnakes/nightly) via deadsnakes in CI if we want to test against that

the first one appears to be testing an implementation detail about the parser (it adds a newline if a file does not add in a newline which is represented in the SyntaxError) -- we can either add a newline to the string being compiled or .rstrip('\n') as suggested above

the second error looks like an actual problem with the line number finding code which probably needs closer attention -- or a bug report on cpython

we should also test if these fail with -X oldparser too as it may indicate a regression with the PEG parser (PEP 617) -- we hit some similar failures in offsets in pyflakes and exposed a few bugs there

asottile · 2020-05-06T19:38:59Z

here's an example patch of something we could do here: https://github.com/asottile/pytest/commit/3366c4f16c7426b2f186cc72d39394f2d139724a

nicoddemus · 2020-05-06T21:14:44Z

@nicoddemus we can probably set up either 3.9-tagged (ppa:deadsnakes/ppa) or 3.9-nightly (ppa:deadsnakes/nightly) via deadsnakes in CI if we want to test against that

Sure thing. We should also make that job an "allowed failure" so the CI doesn't break until 3.9 reaches beta or something like that. Want to try to set this up? 😁

asottile · 2020-05-06T21:20:12Z

github actions doesn't really have a concept of "allowed failures" currently -- (beyond doing it manually with || true and annotating a warning)

or we could wait a week or two, the first beta lands then

nicoddemus · 2020-05-06T21:25:50Z

or we could wait a week or two, the first beta lands then

Fine by me! 👍

hroncok · 2020-05-06T21:48:03Z

The CI job aside, it would be really helpful to figure out if the second failure is a bug in CPython before beta, so beta can contain the fix.

hroncok · 2020-05-07T10:51:05Z

we should also test if these fail with -X oldparser too as it may indicate a regression with the PEG parser (PEP 617) -- we hit some similar failures in offsets in pyflakes and exposed a few bugs there

$ git rev-parse HEAD
81da5dac4840597ed05e2d216a2229acce62b26f

$ tox -e py39 -- -k "test_syntaxerror_rerepresentation or test_getfslineno"
...
FAILED testing/code/test_source.py::test_syntaxerror_rerepresentation - Asser...
FAILED testing/code/test_source.py::test_getfslineno - assert 520 == -1
...

$ head -n1 .tox/py39/bin/pytest   # after editing
#!.../pytest/.tox/py39/bin/python -Xoldparser

$ .tox/py39/bin/pytest -k "test_syntaxerror_rerepresentation or test_getfslineno"
============================= test session starts ==============================
platform linux -- Python 3.9.0a6, pytest-5.4.1.dev235+g81da5dac4, py-1.8.1, pluggy-0.13.1
rootdir: .../pytest, inifile: tox.ini, testpaths: testing
plugins: hypothesis-5.10.5
collected 2820 items / 2818 deselected / 2 selected

testing/code/test_source.py .F                                           [100%]

=================================== FAILURES ===================================
_______________________________ test_getfslineno _______________________________

    def test_getfslineno() -> None:
        def f(x) -> None:
            raise NotImplementedError()
    
        fspath, lineno = getfslineno(f)
    
        assert isinstance(fspath, py.path.local)
        assert fspath.basename == "test_source.py"
        assert lineno == f.__code__.co_firstlineno - 1  # see findsource
    
        class A:
            pass
    
        fspath, lineno = getfslineno(A)
    
        _, A_lineno = inspect.findsource(A)
        assert isinstance(fspath, py.path.local)
        assert fspath.basename == "test_source.py"
        assert lineno == A_lineno
    
        assert getfslineno(3) == ("", -1)
    
        class B:
            pass
    
        B.__name__ = "B2"
>       assert getfslineno(B)[1] == -1
E       assert 520 == -1

testing/code/test_source.py:525: AssertionError
=========================== short test summary info ============================
FAILED testing/code/test_source.py::test_getfslineno - assert 520 == -1
================= 1 failed, 1 passed, 2818 deselected in 3.06s =================

asottile · 2020-05-07T12:21:50Z

ahah ok I figured it out, there was an improvement to inspect.getsource -- I'll send a patch for this since it seems easier to up-front verify now :)

https://bugs.python.org/issue35113

inspect.getsource improved in Python 3.9 (utilizing the ast and __qualname__) See https://bugs.python.org/issue35113 Resolves pytest-dev#7161

hroncok added the type: selftests a problem in the tests of pytest label May 4, 2020

bluetech mentioned this issue May 7, 2020

ci: update github action versions, remove outdated comment #7177

Merged

asottile mentioned this issue May 7, 2020

Fix tests for python3.9 #7179

Merged

hroncok pushed a commit to hroncok/pytest that referenced this issue May 7, 2020

Fix test_getfslineno for Python 3.9

50534fc

inspect.getsource improved in Python 3.9 (utilizing the ast and __qualname__) See https://bugs.python.org/issue35113 Resolves pytest-dev#7161

hroncok pushed a commit to hroncok/pytest that referenced this issue May 7, 2020

Fix test_getfslineno for Python 3.9

ab1996c

inspect.getsource improved in Python 3.9 (utilizing the ast and __qualname__) See https://bugs.python.org/issue35113 Resolves pytest-dev#7161

hroncok mentioned this issue May 7, 2020

[4.6] Fix test_getfslineno for Python 3.9 #7181

Closed

nicoddemus closed this as completed in #7179 May 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

New test failures with Python 3.9.0a6 #7161

New test failures with Python 3.9.0a6 #7161

hroncok commented May 4, 2020 •

edited

Loading

nicoddemus commented May 4, 2020

Uh oh!

hroncok commented May 4, 2020

Uh oh!

asottile commented May 6, 2020

Uh oh!

asottile commented May 6, 2020

Uh oh!

nicoddemus commented May 6, 2020

Uh oh!

asottile commented May 6, 2020

Uh oh!

nicoddemus commented May 6, 2020

Uh oh!

hroncok commented May 6, 2020

Uh oh!

hroncok commented May 7, 2020

Uh oh!

asottile commented May 7, 2020

Uh oh!

Uh oh!

New test failures with Python 3.9.0a6 #7161

New test failures with Python 3.9.0a6 #7161

Comments

hroncok commented May 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nicoddemus commented May 4, 2020

Uh oh!

hroncok commented May 4, 2020

Uh oh!

asottile commented May 6, 2020

Uh oh!

asottile commented May 6, 2020

Uh oh!

nicoddemus commented May 6, 2020

Uh oh!

asottile commented May 6, 2020

Uh oh!

nicoddemus commented May 6, 2020

Uh oh!

hroncok commented May 6, 2020

Uh oh!

hroncok commented May 7, 2020

Uh oh!

asottile commented May 7, 2020

Uh oh!

hroncok commented May 4, 2020 •

edited

Loading