Regression with parallel tests on 2.7.3 and 2.8

[xavfernandez]

Hello,
Some tests in pip test suite started failing with latest pytest releases:

• https://travis-ci.org/pypa/pip/builds/82695909 is develop with parallel testing and pytest==2.8 (2 tests failing)
• https://travis-ci.org/pypa/pip/builds/82716794 is develop but no parallel testing and pytest==2.8 (all green)
• https://travis-ci.org/pypa/pip/builds/82720894 is develop, parallel testing and pytest==2.7.3 (only test_install_from_local_directory_with_symlinks_to_directories failing, test_install_user_in_global_virtualenv_with_conflict_fails ok)
• https://travis-ci.org/pypa/pip/builds/82721613 is develop parallel testing and pytest==2.7.2 (all green)

[xavfernandez]

Bisecting indicated 330de0a as responsible cc @nicoddemus @RonnyPfannschmidt

cf pypa/pip#3341

I'm wondering if https://bitbucket.org/pytest-dev/py/src/b9cdab378c58c3c07dce13ab8bb58463ae6a0bba/py/_path/local.py?at=default&fileviewer=file-view-default#local.py-791:796 should be called with a keep number higher than 3 ?

[RonnyPfannschmidt]

hmm, indeed - i suspect this bug has been lingering longer, the structural change just makes it trigger more easily

[RonnyPfannschmidt]

due to race conditions it seems, that the lockfile creation and the folder creation can intersect badly now, since it got a lot faster - i propose switching to a datetime+pid based folder name in order to reduce the likelihood of collisions

@nicoddemus opinions?

i'd suggest a folder name like 2016-01-04+15:46:33+1337 concatenating date, time and pid

[nicoddemus]

i propose switching to a datetime+pid based folder name in order to reduce the likelihood of collisions

Which folder exactly do you mean @RonnyPfannschmidt? For example, here's what Travis is creating:

 /tmp/pytest-of-travis/pytest-1/popen-gw0/<test-name>

[xavfernandez]

Any new idea ? :)

[nicoddemus]

@RonnyPfannschmidt did you notice my question above? 😁

[xavfernandez]

Any advancement ?
I'd like pip to keep on track with pytest 😿

[RonnyPfannschmidt]

i can put a focus on it next week, this has been delayed due to unrelated personal life events

[xavfernandez]

Any news ? 😿

[xavfernandez]

Gentle ping, I'd rather not have pip indefinitely stranded on pytest==2.7.2 :-/

[RonnyPfannschmidt]

atm i cant promise anything wrt getting at it soonish

[nicoddemus]

@RonnyPfannschmidt could please you explain in more detail why do you think there is a race condition with the lock creation? I ask because looking at make_numbered_dir, the directory creation should avoid any race condition:

            try:
                udir = rootdir.mkdir(prefix + str(maxnum+1))
            except py.error.EEXIST:
                # race condition: another thread/process created the dir
                # in the meantime.  Try counting again
                if lastmax == maxnum:
                    raise
                lastmax = maxnum
                continue

IIUC, the lock file only exists so a directory in use won't be deleted by a new py.test session, but on Travis we only ever have one py.test session active at once (we create a temp directory for each session, with each slave reusing it).

Also, the original change only changed the tmp root directory and I suspect Travis's VMs contains a very small number of files in /tmp, so I don't see how that could affect the outcome of those tests.

Any hints on what's on your mind about the problem might be helpful. 😁

[nicoddemus]

@RonnyPfannschmidt any further thoughts?

Either way, we really should tackle and solve this once and for all in the upcoming pytest sprint, in a couple of weeks.

[RonnyPfannschmidt]

after some more investigation it seems like the tmpdir initialization/re-initialization at slave startup happens multiple times, once with the xdist configuration, once without

[RonnyPfannschmidt]

this is related to #1618

[nicoddemus]

Opened pypa/pip#3863 which fixes this problem, although I couldn't identify why the original code did not work in the first place.

[RonnyPfannschmidt]

my current impression is that the initialization of the tmpdir handler on a remote pytest is incorrect and happens twice due to the option application happening after the pytest initialization instead of before

[xavfernandez]

Well, it's a good news that pip can now use the latest pytest.
But I'm uncomfortable with the fact we don't now why :-/

I guess you've already checked similar logs (PYTEST_DEBUG = 1) but I've recreated a branch to have more infos on the failure: https://travis-ci.org/pypa/pip/jobs/146602684

[nicoddemus]

@xavfernandez agreed, @RonnyPfannschmidt has hunch that the problem might be regarding how options (specially basetemp) is forwarded and handled at the slaves, see #1743.

[RonnyPfannschmidt]

@xavfernandez did some tests showing that my hunch was wrong

[nicoddemus]

@RonnyPfannschmidt I was thinking, while #1743 makes sense, that wouldn't explain why their old tmpdir implementation did not work, after all it was correctly using config._tmpdir_factory (see my PR), which is also used by pytest's own tmpdir.

[RonnyPfannschmidt]

@nicoddemus yes, ever since @xavfernandez demonstrated that the issue is in fact not related to that, i do wonder what is the actual cause