Lines incorrectly reporting as not covered when running with --branch

[nedbat]

Originally reported by David MacIver (Bitbucket: david_maciver_, GitHub: Unknown)

---

I ran into this problem when testing Hypothesis: Lines that are definitely covered by a test are showing up as uncovered in coverage.

The following is the test file:

#!python

# coding=utf-8

from __future__ import division, print_function, absolute_import

import hypothesis.strategies as st


def test_very_deep_deferral():
    def strat(i):
        if i == 0:
            return st.deferred(lambda: st.one_of(strategies + [st.none()]))
        else:
            return st.deferred(
                lambda: st.tuples(strategies[(i + 1) % len(strategies)]))

    strategies = list(map(strat, range(100)))

    assert strategies[0].has_reusable_values
    assert not strategies[0].is_empty


if __name__ == '__main__':
    test_very_deep_deferral()

When testing Hypothesis commit 8844ca758737a6439a82507e7910972c6fa22358 (bb), this shows lines 156-158 in src/hypothesis/searchstrategy/strategies.py as uncovered. However the lines are definitely covered by this test (see below for evidence).

To reproduce this problem:

git clone [email protected]:HypothesisWorks/hypothesis-python.git
cd hypothesis-python
git checkout [8844ca758737a6439a82507e7910972c6fa22358 (bb)](https://bitbucket.org/ned/coveragepy/commits/8844ca758737a6439a82507e7910972c6fa22358)

(now copy the above example as test_local.py)

PYTHONPATH=src python -m coverage run --branch --rcfile=/dev/null --include=src/hypothesis/searchstrategy/strategies.py test_local.py
coverage report --rcfile=/dev/null --show-missing

Salient features:

• The lines 156-158 show up in the report.
• If you rerun it without "--branch" in the "coverage run" line they do not.
• If you add "assert False" at line 156, the test fails. Thus the coverage run without "--branch" is correct.

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/605

[nedbat]

Thanks for the detailed instructions. I had to install a bunch of requirements (including enum34 manually, which isn't mentioned in any requirements file?). But then I did not see the results you see:

$ PYTHONPATH=src python -m coverage run --branch --rcfile=/dev/null --include=src/hypothesis/searchstrategy/strategies.py test_local.py
$ coverage report --rcfile=/dev/null --show-missing
Name                                          Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------------------------------
src/hypothesis/searchstrategy/strategies.py     244    116     68      3    46%   37-40, 143, 170-173, 212, 225-267, 278, 290-291, 304, 311, 320-322, 330-339, 342, 345, 370-371, 384-402, 405-414, 417, 420-421, 425-432, 445-448, 451, 454-459, 462, 467, 471-478, 482, 491-493, 496, 499-504, 507, 510-526, 530-534, 133->143, 369->370, 377->exit
$ PYTHONPATH=src python -m coverage run --rcfile=/dev/null --include=src/hypothesis/searchstrategy/strategies.py test_local.py
$ coverage report --rcfile=/dev/null --show-missing
Name                                          Stmts   Miss  Cover   Missing
---------------------------------------------------------------------------
src/hypothesis/searchstrategy/strategies.py     244    116    52%   37-40, 143, 170-173, 212, 225-267, 278, 290-291, 304, 311, 320-322, 330-339, 342, 345, 370-371, 384-402, 405-414, 417, 420-421, 425-432, 445-448, 451, 454-459, 462, 467, 471-478, 482, 491-493, 496, 499-504, 507, 510-526, 530-534

[bluefish6]

Hi,

I was able to get probably the same bug with a minimal example with pytest:

test_foo.py

def test_literal_list():
    assert(
        [0,1,2] == [0,1,2]
    )

def test_foobar():
    assert(
        [0,1,2] == list(x for x in range(3))
    )

def test_foobar2():
    assert (
        [x for x in range(3)] == [0, 1, 2]
    )

def test_foobar2_oneline():
    assert ([x for x in range(3)] == [0, 1, 2])

pytest --cov-branch --cov-report term-missing

(which more or less translates to coverage report -m with branch coverage)

Results:

----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name                        Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------------

test_foo.py                     8      0      6      2    86%   7->exit, 12->exit

As you can see, the first test was "covered" in full. The test_foobar and test_foobar2 seem to be covered partially, and test_foobar2_oneline which is the same as test_foobar2, but without splitting this into multiple lines, is "covered" in full.

Also, I made another test - the same file with executing the test functions manually and without pytest at all:

def test_literal_list():
    assert(
        [0,1,2] == [0,1,2]
    )

def test_foobar():
    assert(
        [0,1,2] == list(x for x in range(3))
    )

def test_foobar2():
    assert (
        [x for x in range(3)] == [0, 1, 2]
    )

def test_foobar2_oneline():
    assert ([x for x in range(3)] == [0, 1, 2])

test_literal_list()
test_foobar()
test_foobar2()
test_foobar2_oneline()

coverage run --branch test_foo.py
coverage report -m
Name          Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------------
test_foo.py      12      0      6      0   100%

Note that:

• everything is covered (which could indicate some issue with the way pytest is executing coverage)
• there are 6 branches, which seems kind of unlikely, as I don't have any conditional blocks or loops.

Coverage 4.5.1 (I had the same issue with 4.0, but with different result presentation 7->-8 instead of 7->exit)

pytest 3.5.0
pytest-cov 2.5.1

[nedbat]

@bluefish6 We should probably create this as a new bug, in case it is not the same cause as the original report here.

But I tried running your code, and did not see the same results. pytest-cov didn't produce any reporting output:

$ pytest --cov-branch --cov-report term-missing
=================================== test session starts ====================================
platform darwin -- Python 3.6.6, pytest-3.5.0, py-1.5.4, pluggy-0.6.0
rootdir: /Users/ned/coverage/bug605, inifile:
plugins: cov-2.5.1
collected 4 items

test_foo.py ....                                                                     [100%]

================================= 4 passed in 0.01 seconds =================================

When I run with coverage directly, I get 100% coverage:

$ coverage run --source=. -m py.test
=================================== test session starts ====================================
platform darwin -- Python 3.6.6, pytest-3.5.0, py-1.5.4, pluggy-0.6.0
rootdir: /Users/ned/coverage/bug605, inifile:
plugins: cov-2.5.1
collected 4 items

test_foo.py ....                                                                     [100%]

================================= 4 passed in 0.01 seconds =================================

$ coverage report -m
Name          Stmts   Miss  Cover   Missing
-------------------------------------------
test_foo.py       8      0   100%

The code I ran:

$ cat test_foo.py
def test_literal_list():
    assert(
        [0,1,2] == [0,1,2]
    )

def test_foobar():
    assert(
        [0,1,2] == list(x for x in range(3))
    )

def test_foobar2():
    assert (
        [x for x in range(3)] == [0, 1, 2]
    )

def test_foobar2_oneline():
    assert ([x for x in range(3)] == [0, 1, 2])

$ python -V
Python 3.6.6

$ pip freeze
atomicwrites==1.1.5
attrs==18.1.0
coverage==4.5.1
more-itertools==4.2.0
pluggy==0.6.0
py==1.5.4
pytest==3.5.0
pytest-cov==2.5.1
six==1.11.0

[a-johnston]

Hi,

I was seeing similar behavior where lambdas being passed to a method were being reported as a missed arc in python 2.7.6, python 3.5.1 and python 3.6.0 (if that matters). In my case, changing the lambdas to the appropriate partial caused the coverage check to pass. Interestingly, in a chain of method calls in which two lambdas are passed, the same line (ie 47->exit, 47->exit) is reported twice.

I don't have time right now but I'll take a closer look later today.

[bluefish6]

@nedbat
Hi Ned,

Thanks for taking a look!

I'm terribly sorry - I mistakenly did not include the parameter --cov=..
It should be:

py.test --cov=. --cov-report term-missing --cov-branch
================================== test session starts ==================================
platform linux -- Python 3.5.2, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/antek/praca/evox/coverage_test, inifile:
plugins: cov-2.5.1
collected 4 items                                                                       

test_foo.py ....                                                                  [100%]

----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name          Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------------
test_foo.py      12      0      6      2    89%   7->exit, 12->exit


=============================== 4 passed in 0.03 seconds ================================

I was able to reproduce it also with:

coverage run --source=. -m py.test --cov=. --cov-branch --cov-report term-missing
================================== test session starts ==================================
platform linux -- Python 3.5.2, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/antek/praca/evox/coverage_test, inifile:
plugins: cov-2.5.1
collected 4 items                                                                       

test_foo.py ....                                                                  [100%]

----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name          Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------------
test_foo.py      12      0      6      2    89%   7->exit, 12->exit


=============================== 4 passed in 0.04 seconds ================================
Coverage.py warning: No data was collected. (no-data-collected)

I solved the mystery. The one to blame is pytest and it's "assertion rewrite hook". As far as I understand, it changes the code of all asserts to be a special fancy assert-like function that shows better errors. The problem is, that for some reason it breaks the coverage for cases like:

    assert (
        [x for x in range(3)] == [0, 1, 2]
    )

Quick test - let's pretend that our system does not support the hook: (.../site-packages/_pytest/assertion/__init.__.py) and always raise the Jython-dependent error:

def install_importhook(config):
    """Try to install the rewrite hook, raise SystemError if it fails."""
    raise SystemError("rewrite not supported") # <-- add this to force an Exception to not install the hook
    # Jython has an AST bug that make the assertion rewriting hook malfunction.
    if sys.platform.startswith("java"):
        raise SystemError("rewrite not supported")

And voilla, no branches uncovered.

So to sum up - the bug is somewhere inside the assertion rewrite mechanism of pytest. I will open an issue there with link to this issue for reference.

[bluefish6]

Issue created in pytest repo: pytest-dev/pytest#3689

[bluefish6]

Possibly my bug is a duplicate of #515

[ssbarnea]

Sadly this 5 year old bug is still valid and probably it would worth even a runtime warning for those that make the mistake of enabling branch coverage.

It has nothing to do with pytest, as I had the code below that also reports no branch coverage, even if I tested with the debugger to see that the loop really executes.

      if isinstance(when, list):
          for item in when:
              return _changed_in_when(item)

[nedbat]

@ssbarnea can you provide a complete example?

[ssbarnea]

I will to isolate it after I deal with current pile of incoming changes. You can see that mentioned disablement that increases total coverage ~4% by disabling branch coverage. I know that branch is good, but is good only when it works.

I wonder how hard it will be to reproduce it in isolation, as we do use xdist and subprocess coverage commands too, I would not be surprised to discover that the issue happens only in conjunction with one of these.

[barraponto]

@ssbarnea I saw the same issue with branch, using next("comprehension with if", None). Turns out coverage was correct: i wasn't testing the None case.

In your case, you may be assuming _changed_in_when is called, but you should consider the possibility it isn't (namely, if when is an empty iterable).