xdist: Different tests were collected

[mhils]

Steps to reproduce the problem:

1. Create some test cases that look as follows:

   def _test(a):
       def t(self):
           assert a == a
       return t
   
   class TestFoo(object):
       test_x = _test("a")
       test_y = _test("b")

2. Run py.test -n 4

What is the expected behavior?

The tests should pass.

What went wrong?

λ py.test test_foo.py -n 4
============================= test session starts =============================
platform win32 -- Python 3.5.0, pytest-2.8.0, py-1.4.30, pluggy-0.3.1
rootdir: C:\Users\user\git\netlib, inifile:
plugins: cov-2.1.0, xdist-1.13.1
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
collecting 0 items / 3 errors
=================================== ERRORS ====================================
____________________________ ERROR collecting gw0 _____________________________
Different tests were collected between gw2 and gw0. The difference is:
--- gw2

+++ gw0

@@ -1,2 +1,2 @@

+test_foo.py::TestFoo::()::test_y
 test_foo.py::TestFoo::()::test_x
-test_foo.py::TestFoo::()::test_y
=========================== 3 error in 1.17 seconds ===========================

This is the minimal test case I could break it down to. Looks like there's an issue if we're not defing functions. Any help would be appreciated! 😃

[RonnyPfannschmidt]

the easy way could be, to declare the python hash seed, so the unordered parts of the collection have a high chance for having the same order

the problem is, that when you use test items that have a indistinguishable source code location, order is left to mappings, and mapping order is undefined

alternatively you could use a type whose items are put into a ordereddict

[AbdealiLoKo]

As @RonnyPfannschmidt mentioned about the hash seed,
This gets solved for me if I use the pytest-env plugin to do:

env =
    PYTHONHASHSEED=0

[RonnyPfannschmidt]

Im closing this one AS invalid, please reopened if a New Detail comes up

[trtg]

This is still a problem for me, I have not been able to find a single project where I can successfully use more than 1 core without encountering a failure. I looked through the list of projects that use py.test and tried cloning a bunch of them and running py.test -n 2. For example tiddlyweb from here: git@github.com:tiddlyweb/tiddlyweb.git

if that project is tested with py.test -n 2 test/
3 failures appear:
______________________________________________________ ERROR collecting gw0 _______________________________________________________
Different tests were collected between gw1 and gw0. The difference is:

This happened on the pypy project and on the project I'm currently working on that I want to test faster (the test suite is currently 24 minutes long). So far xdist has been unusable for me, since specifying more than one core guarantees failure. Are there more special considerations to keep in mind if I want to parallelize with py.test -n 2 ?

[RonnyPfannschmidt]

@trtg the project you linked uses a demonstrably broken approach to generate tests - thats pretty much its own fault

[trtg]

Did the exploration of changing how tests are distributed discussed here:
#175
ever move forward?

Pierre-yves Rofes changes that allow assigning all the tests in one class
to the same core would be perfect for my use case.

On Sep 13, 2016 11:57 PM, "Ronny Pfannschmidt" notifications@github.com
wrote:

@trtg https://github.com/trtg the project you linked uses a demonstrably
broken approach to generate tests - thats pretty much its own fault

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1075 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQk-BiY2We9X391FZo_YGY9CZcclzTJks5qp5rlgaJpZM4GEYrm
.

[RonnyPfannschmidt]

@trtg the discussionadvanced in the xdist issue tracker, but no implementation did take place

[nicoddemus]

For the record, the discussion took place in pytest-dev/pytest-xdist#18.