Parametrization order changes between py2.7 and py3

[mshriver]

I am observing different parametrization order on the same test repository and command syntax between py2.7 and py3.6.6

The source repository where this is observed is ManageIQ/integration tests, and involves a tree of fixtures and parametrization. I'll attempt to create a 'unit' level recreation to simplify, but wanted to report ASAP. I've discussed this a bit with @RonnyPfannschmidt and he is familiar with integration_tests and its environment.

In python 2.7, I observe the following test function name and param ids with the linked test/fixture:
test_migration_long_name[virtualcenter-6.0-rhevm-4.2]

In python 3.6.6, the two parameters have switched:
test_migration_long_name[rhevm-4.2-virtualcenter-6.0]

Fixture with first two parameters:
https://github.com/ManageIQ/integration_tests/blob/master/cfme/fixtures/v2v.py#L38

Test module with provider and second_provider parametrized:
https://github.com/ManageIQ/integration_tests/blob/master/cfme/tests/v2v/test_v2v_migrations.py#L19

Example test function above from same module, line 651:
https://github.com/ManageIQ/integration_tests/blob/master/cfme/tests/v2v/test_v2v_migrations.py#L651

OS: Fedora 27
Both py.test runs are done in a virtualenv with frozen pip.
pytest version 3.4.1
Frozen requirements for py2.7 and py3.6.6 environments:
py2 requirements
py3 requirements

[mshriver]

@RonnyPfannschmidt @nicoddemus

I'm sure I'm not the only one looking at the looming py2 end-of-life that is starting to encounter issues with these differences between the versions.

The test collection order ends up impacting other parts of my team's toolchain, and while the specific instance of parameter switching does not break our testing, it is certainly not expected, and keeps us from running both environments together while we try to migrate to py3.

Can I get a sense from you guys what your plan is when dealing with issues like this? As long as collection itself is not impacted in py3, given the EOL for py2 I could see your decision being to ignore such an issue. If that is the case, I need to plan my team's migration to py3 accordingly.

Thanks!

[nicoddemus]

Hi @mshriver,

Sorry for the silence on this, I must have forgotten to come back to this after applying the labels.

If you can produce a reproducible example, it would be great. It seems like a bug, we should be creating the same parametrization on py 2 and py3.

Can I get a sense from you guys what your plan is when dealing with issues like this?

Sure, we have written it down here. Let me know if there are any questions.

[davidfyu]

Hi @mshriver,

I would like to try tackling this issue. For reference, I am a university student and must complete an open source contribution for an upcoming assignment.

My understanding is that for this issue, we should simply start at the source of where the error is observed (ManageIQ/integration_tests) and make the necessary parameter changes. After building the tests in 2.7 and 3.6.6, the subsequent failures would indicate which files need to be changed in the “tree of fixtures.” Is this correct?

Thank you!

[blueyed]

@davidfyu
Thanks for considering pytest for your contribution.
Regarding this issue it is more about fixing pytest to keep the same ordering here, with the given input (ManageIQ/integration_tests).
To get this started I suggest looking at the output of --collect-only, which should show the different order then already.
The next step would be to extract this into a pytest test (using testdir fixture), and then test it using tox -e py27 and tox -e py36.

[asottile]

hello, first off thank you for the issue!

python 2.x support has ended for pytest core.

we've decided in #7296 to close the python-2-specific issues to free up some space in our backlog. however, in accordance to our python 2.7 and 3.4 support community patches will still be accepted to the 4.6 series to fix bugs for python 2. (so if this issue is important to you and you have a patch to fix it, feel free to make a PR targeting the 4.6.x branch despite this ticket being closed).