Test pytest 3.1 and master on CI

[nicoddemus]

Also use 3.6 preferably in general

This supersedes #166

[nicoddemus]

Not sure if this is a good idea or not, thought I would open the PR directly instead of raising an issue;

[RonnyPfannschmidt]

the one for pytestmaster should be a own stage and use a cronjob

[nicoddemus]

test_looponfail_removed_test is failing in pytest 3.1:

=================================== FAILURES ===================================
________________ TestLooponFailing.test_looponfail_removed_test ________________

self = <test_looponfail.TestLooponFailing instance at 0x7fcfd454e248>
testdir = <Testdir local('/tmp/pytest-of-travis/pytest-3/testdir/test_looponfail_removed_test0')>

    def test_looponfail_removed_test(self, testdir):
        modcol = testdir.getmodulecol("""
                def test_one():
                    assert 0
                def test_two():
                    assert 0
            """)
        remotecontrol = RemoteControl(modcol.config)
        remotecontrol.loop_once()
        assert len(remotecontrol.failures) == 2
    
        modcol.fspath.write(py.code.Source("""
                def test_xxx(): # renamed test
                    assert 0
                def test_two():
                    assert 1 # pass now
            """))
        removepyc(modcol.fspath)
        remotecontrol.loop_once()
>       assert len(remotecontrol.failures) == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = len(['test_looponfail_removed_test.py::test_xxx'])
E        +    where ['test_looponfail_removed_test.py::test_xxx'] = <xdist.looponfail.RemoteControl object at 0x7fcfd4c63b50>.failures

/home/travis/build/nicoddemus/pytest-xdist/testing/test_looponfail.py:215: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/pytest-of-travis/pytest-3/testdir/test_looponfail_removed_test0, inifile:
plugins: xdist-1.16.1.dev39+ga236b9a
collected 0 items

=============================== warnings summary ===============================
None
  Module already imported so can not be re-written: xdist
  Module already imported so can not be re-written: xdist
  Module already imported so can not be re-written: xdist

-- Docs: http://doc.pytest.org/en/latest/warnings.html
========================== 3 warnings in 0.00 seconds ==========================
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/pytest-of-travis/pytest-3/testdir/test_looponfail_removed_test0, inifile:
plugins: xdist-1.16.1.dev39+ga236b9a
collected 2 items
collected 2 items

test_looponfail_removed_test.py FF

=================================== FAILURES ===================================
___________________________________ test_one ___________________________________

    def test_one():
>       assert 0
E       assert 0

test_looponfail_removed_test.py:2: AssertionError
___________________________________ test_two ___________________________________

    def test_two():
>       assert 0
E       assert 0

test_looponfail_removed_test.py:4: AssertionError
=========================== 2 failed in 0.03 seconds ===========================
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/pytest-of-travis/pytest-3/testdir/test_looponfail_removed_test0, inifile:
plugins: xdist-1.16.1.dev39+ga236b9a
collected 7 items
collected 7 items

test_looponfail_removed_test.py F.

=================================== FAILURES ===================================
___________________________________ test_xxx ___________________________________

    def test_xxx(): # renamed test
>       assert 0
E       assert 0

test_looponfail_removed_test.py:3: AssertionError
====================== 1 failed, 1 passed in 0.03 seconds ======================
==== 1 failed, 94 passed, 5 skipped, 10 xfailed, 1 xpassed in 37.43 seconds ====

But I don't understand the test itself.

It first runs this with loop on failure:

                def test_one():
                    assert 0
                def test_two():
                    assert 0

This fails with two tests. Nice, as expected.

Then it changes the code to:

                def test_xxx(): # renamed test
                    assert 0
                def test_two():
                    assert 1 # pass now

And expects no failures... but test_xxx clearly fails now, so to me it looks like the test is wrong.

But test_looponfail_removed_test passes in other pytest versions, so I'm not sure what's supposed to be happening here.

I never looked into the internals of how the loop-on-failure work, so sorry if it is obvious why test_looponfail_removed_test should behave like it does in previous pytest versions.

[timyhou]

I took a look at this and full-disclosure I have no idea how the loop-on-failure is supposed to behave.

What I seem to observe is that in pytest <= 3.0.7 after the test script is modified, it gathered the "known" tests first and ran them. The next loop_once() would gather any new tests and run them with the previous ones.
In 3.1.2. what seems to be happening, is that pytest doesn't seem to recognize test_two() as part of the initial set of tests so the loop thinks it has two brand new tests and runs both of them. I feel the test is not completely invalid because test_looponfail_from_one_to_two_tests still works in pytest-3.1.2 where we can see the similar behavior but "adding" instead of "removing."