New test failures with Python 3.9.0a6

[hroncok]

After upgrading to Python 3.9.0a6, I see 2 new test failures, both in testing/code/test_source.py.

This is Fedora 32, CPython 3.9.0a6, with tox -e py39, master branch @ 80e5098.

$ .tox/py39/bin/pip list
Package          Version                
---------------- -----------------------
argcomplete      1.11.1                 
attrs            19.3.0                 
certifi          2020.4.5.1             
chardet          3.0.4                  
elementpath      1.4.4                  
hypothesis       5.10.5                 
idna             2.9                    
mock             4.0.2                  
more-itertools   8.2.0                  
nose             1.3.7                  
packaging        20.3                   
pip              19.3.1                 
pluggy           0.13.1                 
py               1.8.1                  
Pygments         2.6.1                  
pyparsing        2.4.7                  
pytest           5.4.1.dev171+g80e509840
requests         2.23.0                 
setuptools       41.6.0                 
six              1.14.0                 
sortedcontainers 2.1.0                  
urllib3          1.25.9                 
wcwidth          0.1.9                  
wheel            0.33.6                 
xmlschema        1.1.3       

• a detailed description of the bug or suggestion
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

$ tox -e py39 -- -k "test_syntaxerror_rerepresentation or test_getfslineno"
py39 inst-nodeps: .../pytest/.tox/.tmp/package/1/pytest-5.4.1.dev171+g80e509840.tar.gz
py39 installed: argcomplete==1.11.1,attrs==19.3.0,certifi==2020.4.5.1,chardet==3.0.4,elementpath==1.4.4,hypothesis==5.10.5,idna==2.9,mock==4.0.2,more-itertools==8.2.0,nose==1.3.7,packaging==20.3,pluggy==0.13.1,py==1.8.1,Pygments==2.6.1,pyparsing==2.4.7,pytest==5.0.2.dev41+g183750fa8,requests==2.23.0,six==1.14.0,sortedcontainers==2.1.0,urllib3==1.25.9,wcwidth==0.1.9,xmlschema==1.1.3
py39 run-test-pre: PYTHONHASHSEED='3233708815'
py39 run-test: commands[0] | pytest -k 'test_syntaxerror_rerepresentation or test_getfslineno'
============================= test session starts ==============================
platform linux -- Python 3.9.0a6, pytest-5.4.1.dev171+g80e509840, py-1.8.1, pluggy-0.13.1
cachedir: .tox/py39/.pytest_cache
rootdir: .../pytest, inifile: tox.ini, testpaths: testing
plugins: hypothesis-5.10.5
collected 2688 items / 2686 deselected / 2 selected

testing/code/test_source.py FF                                           [100%]

=================================== FAILURES ===================================
______________________ test_syntaxerror_rerepresentation _______________________

    def test_syntaxerror_rerepresentation() -> None:
        ex = pytest.raises(SyntaxError, _pytest._code.compile, "xyz xyz")
        assert ex is not None
        assert ex.value.lineno == 1
        assert ex.value.offset in {5, 7}  # cpython: 7, pypy3.6 7.1.1: 5
>       assert ex.value.text == "xyz xyz\n"
E       AssertionError: assert 'xyz xyz' == 'xyz xyz\n'
E         - xyz xyz
E         ?        -
E         + xyz xyz

testing/code/test_source.py:125: AssertionError
_______________________________ test_getfslineno _______________________________

    def test_getfslineno() -> None:
        def f(x) -> None:
            raise NotImplementedError()
    
        fspath, lineno = getfslineno(f)
    
        assert isinstance(fspath, py.path.local)
        assert fspath.basename == "test_source.py"
        assert lineno == f.__code__.co_firstlineno - 1  # see findsource
    
        class A:
            pass
    
        fspath, lineno = getfslineno(A)
    
        _, A_lineno = inspect.findsource(A)
        assert isinstance(fspath, py.path.local)
        assert fspath.basename == "test_source.py"
        assert lineno == A_lineno
    
        assert getfslineno(3) == ("", -1)
    
        class B:
            pass
    
        B.__name__ = "B2"
>       assert getfslineno(B)[1] == -1
E       assert 520 == -1

testing/code/test_source.py:525: AssertionError
=========================== short test summary info ============================
FAILED testing/code/test_source.py::test_syntaxerror_rerepresentation - Asser...
FAILED testing/code/test_source.py::test_getfslineno - assert 520 == -1
====================== 2 failed, 2686 deselected in 3.01s ======================
ERROR: InvocationError for command .../pytest/.tox/py39/bin/pytest -k 'test_syntaxerror_rerepresentation or test_getfslineno' (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py39: commands failed

[nicoddemus]

Thanks @hroncok!

We will take a look at this once we can get some version of Python 3.9 running on CI.

[hroncok]

I think the first failure might be "fixed" by an rstrip call, however I haven't really grasped the second one :(

[asottile]

@nicoddemus we can probably set up either 3.9-tagged (ppa:deadsnakes/ppa) or 3.9-nightly (ppa:deadsnakes/nightly) via deadsnakes in CI if we want to test against that

the first one appears to be testing an implementation detail about the parser (it adds a newline if a file does not add in a newline which is represented in the SyntaxError) -- we can either add a newline to the string being compiled or .rstrip('\n') as suggested above

the second error looks like an actual problem with the line number finding code which probably needs closer attention -- or a bug report on cpython

we should also test if these fail with -X oldparser too as it may indicate a regression with the PEG parser (PEP 617) -- we hit some similar failures in offsets in pyflakes and exposed a few bugs there

[asottile]

here's an example patch of something we could do here: https://github.com/asottile/pytest/commit/3366c4f16c7426b2f186cc72d39394f2d139724a

[nicoddemus]

@nicoddemus we can probably set up either 3.9-tagged (ppa:deadsnakes/ppa) or 3.9-nightly (ppa:deadsnakes/nightly) via deadsnakes in CI if we want to test against that

Sure thing. We should also make that job an "allowed failure" so the CI doesn't break until 3.9 reaches beta or something like that. Want to try to set this up? 😁

[asottile]

github actions doesn't really have a concept of "allowed failures" currently -- (beyond doing it manually with || true and annotating a warning)

or we could wait a week or two, the first beta lands then

[nicoddemus]

or we could wait a week or two, the first beta lands then

Fine by me! 👍

[hroncok]

The CI job aside, it would be really helpful to figure out if the second failure is a bug in CPython before beta, so beta can contain the fix.

[hroncok]

we should also test if these fail with -X oldparser too as it may indicate a regression with the PEG parser (PEP 617) -- we hit some similar failures in offsets in pyflakes and exposed a few bugs there

$ git rev-parse HEAD
81da5dac4840597ed05e2d216a2229acce62b26f

$ tox -e py39 -- -k "test_syntaxerror_rerepresentation or test_getfslineno"
...
FAILED testing/code/test_source.py::test_syntaxerror_rerepresentation - Asser...
FAILED testing/code/test_source.py::test_getfslineno - assert 520 == -1
...

$ head -n1 .tox/py39/bin/pytest   # after editing
#!.../pytest/.tox/py39/bin/python -Xoldparser

$ .tox/py39/bin/pytest -k "test_syntaxerror_rerepresentation or test_getfslineno"
============================= test session starts ==============================
platform linux -- Python 3.9.0a6, pytest-5.4.1.dev235+g81da5dac4, py-1.8.1, pluggy-0.13.1
rootdir: .../pytest, inifile: tox.ini, testpaths: testing
plugins: hypothesis-5.10.5
collected 2820 items / 2818 deselected / 2 selected

testing/code/test_source.py .F                                           [100%]

=================================== FAILURES ===================================
_______________________________ test_getfslineno _______________________________

    def test_getfslineno() -> None:
        def f(x) -> None:
            raise NotImplementedError()
    
        fspath, lineno = getfslineno(f)
    
        assert isinstance(fspath, py.path.local)
        assert fspath.basename == "test_source.py"
        assert lineno == f.__code__.co_firstlineno - 1  # see findsource
    
        class A:
            pass
    
        fspath, lineno = getfslineno(A)
    
        _, A_lineno = inspect.findsource(A)
        assert isinstance(fspath, py.path.local)
        assert fspath.basename == "test_source.py"
        assert lineno == A_lineno
    
        assert getfslineno(3) == ("", -1)
    
        class B:
            pass
    
        B.__name__ = "B2"
>       assert getfslineno(B)[1] == -1
E       assert 520 == -1

testing/code/test_source.py:525: AssertionError
=========================== short test summary info ============================
FAILED testing/code/test_source.py::test_getfslineno - assert 520 == -1
================= 1 failed, 1 passed, 2818 deselected in 3.06s =================

[asottile]

ahah ok I figured it out, there was an improvement to inspect.getsource -- I'll send a patch for this since it seems easier to up-front verify now :)

https://bugs.python.org/issue35113