Can't run function/method that uses parametrization

[sloria]

To reproduce:

Given the following code:

import pytest

@pytest.mark.parametrize(('inp', 'expected'),
[
    ('2 + 2', 4),
    ('0 + 0', 0),
    ('2 + 2.0', 4.0),
])
def test_adding(inp, expected):
    assert eval(inp) == expected

Executing :Pytest function verbose results in the following error.

============================================================================================ test session starts ============================================================================================
platform darwin -- Python 3.4.2 -- py-1.4.26 -- pytest-2.6.4 -- /Users/sloria/miniconda/envs/fresh/bin/python3
plugins: xdist
collecting ... 
=============================================================================================  in 0.01 seconds ==============================================================================================
ERROR: not found: /Users/sloria/projects/python-projects/t.py::test_adding
(no name '/Users/sloria/projects/python-projects/t.py::test_adding' in any of [<Module 't.py'>])

Note: :Pytest file works fine. However, it'd be very nice to run paramerized test functions and methods in isolation.

[alfredodeza]

On first look, this seems to be a bug on py.test itself. py.test uses a notation for specifying what method/class/function to run (as seen in the error output).

And trying to replicate this, the first thing I tried was to point py.test to the function on the command line itself but was unable to do so.

Did you find a way to run that specific test function from the command line using that path::function notation?

[alfredodeza]

Aha! so, py.test will change the node ID of the test of parametrized functions/methods but it does this at runtime, by calculating your parametrization values.

In a test_fail.py with your pasted example, this is how the collection of tests look:

============================== test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- /Users/alfredo/.virtualenvs/tmp/bin/python
collected 3 items
<Module 'test_fail.py'>
  <Function 'test_adding[2 + 2-4]'>
  <Function 'test_adding[0 + 0-0]'>
  <Function 'test_adding[2 + 2.0-4.0]'>

===============================  in 0.00 seconds ================================

Each of those functions is a node defining its own id with a changed name.

And I am able to run a single of those by specifying it like:

(tmp)papaya-2 ~/tmp ᓆ py.test -vvv "/Users/alfredo/tmp/test_fail.py::test_adding[2 + 2-4]"
============================== test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- /Users/alfredo/.virtualenvs/tmp/bin/python
collected 4 items

test_fail.py:4: test_adding[2 + 2-4] PASSED

=========================== 1 passed in 0.01 seconds ============================

I will have to open an issue on the py.test tracker so that they can see if there could be a way to tell the engine to generically use all the parametrizations available for that given test.

[alfredodeza]

Opened an issue on py.test tracker: https://bitbucket.org/hpk42/pytest/issue/649/parametrized-test-nodes-cannot-be

[sloria]

Would it be possible to select the test based on a substring match? http://pytest.org/latest/example/markers.html#using-k-expr-to-select-tests-based-on-their-name

[alfredodeza]

Yeah, it seems like a combination of both will do the trick. This will take a bit of effort to put together though :( We need to correctly detect if we are dealing with a method or function so we can use as much information as possible and avoid selecting more than one test (which can happen with -k)

[alfredodeza]

@sloria I just pushed an attempt to support this, and I was able to run parametrized functions and methods (including your test example).

However, and specifically for functions, this may cause more tests to run than desired if for example, you have a test function called test_adding and some other test function (or test method) that starts with test_adding, like test_adding_other.

Since -k matches the string, this means that non-unique test names that start the same will cause to be run.

Would you mind trying the https://github.com/alfredodeza/pytest.vim/tree/issue23 branch and test it out?

[sloria]

Thanks for the patch. I will try it tomorrow when I have some free time.

[sloria]

@alfredodeza The patch works as expected. I was able to run parametrized tests, but tests whose names begin with the same substring also run. For me, this is fine for the time being until the pytest issue you reported is addressed; better to have too many tests run than none. =)

[sloria]

To amend my previous comment: I ended up reverting back to master... turns out I name my test functions with the same prefix more often than I thought.

Hopefully there's a better solution that doesn't add too much complexity.

[alfredodeza]

Even though that is the case for you, I think this plugin should still try to work with parametrized tests vs. not handling them at all. I will merge this into master shortly unless you have strong opinion against it?

I've been bit by this problem before and I rather have a few extra tests run (in the worse case scenario) than just a generic error.

[sloria]

@alfredodeza I think it is fine to merge. I have no strong opinion either way.

[asafpelegcodes]

If I did a pip install --upgrade will I be able to see the bug fixed, I'm currently experiencing the same issue

[alfredodeza]

@asafpelegcodes this issue is from py.test and has not been fixed. This branch fixes this problem in the sense that it applied a filter option (with -k) so you should be able to run parametrized tests and not see an error.

py.test issue is here: https://bitbucket.org/hpk42/pytest/issue/649/parametrized-test-nodes-cannot-be