pytest-cov seems to override the use of [paths] in .coveragerc, probably because it passes `source=` to the coverage API

[ceridwen]

The code for my project, including tox.ini and .coveragerc, is at https://anonscm.debian.org/cgit/reproducible/reprotest.git/tree/?h=variations . This is on Ubuntu Linux 16.04.

I set up these tests using coverage, and now I'm trying to get subprocess coverage using pytest-cov. There's a lot of output here, unfortunately, but I'm afraid trimming it will obscure what's happening. For each run, I'm running tox -r -- -s -k simple_builds[null] and then looking at the terminal output of coverage report.

Running commands = coverage run --parallel -m py.test {posargs} tests/tests.py, which is what I was using before adding pytest-cov:

.tox/py35/lib/python3.5/site-packages/_pytest/__init__.py                                   1      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/_pytest/_argcomplete.py                              33     23     14      1    28%
.tox/py35/lib/python3.5/site-packages/_pytest/_code/__init__.py                             8      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/_pytest/_code/code.py                               565    433    188      5    18%
.tox/py35/lib/python3.5/site-packages/_pytest/_code/source.py                             287    246    126      0    10%
.tox/py35/lib/python3.5/site-packages/_pytest/_pluggy.py                                    6      3      0      0    50%
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/__init__.py                        69     21     29      6    55%
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py                        557    208    188     35    60%
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/util.py                           201    178    114      0     7%
.tox/py35/lib/python3.5/site-packages/_pytest/cacheprovider.py                            139     70     61     13    43%
.tox/py35/lib/python3.5/site-packages/_pytest/capture.py                                  324    156     98     31    48%
.tox/py35/lib/python3.5/site-packages/_pytest/compat.py                                   113     55     42     12    46%
.tox/py35/lib/python3.5/site-packages/_pytest/config.py                                   837    316    360     59    56%
.tox/py35/lib/python3.5/site-packages/_pytest/debugging.py                                 88     55     18      1    32%
.tox/py35/lib/python3.5/site-packages/_pytest/deprecated.py                                 6      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/_pytest/doctest.py                                  174    128     50      2    21%
.tox/py35/lib/python3.5/site-packages/_pytest/fixtures.py                                 651    197    226     45    66%
.tox/py35/lib/python3.5/site-packages/_pytest/freeze_support.py                            19     15      8      0    15%
.tox/py35/lib/python3.5/site-packages/_pytest/helpconfig.py                                96     70     32      5    24%
.tox/py35/lib/python3.5/site-packages/_pytest/hookspec.py                                  51      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/_pytest/junitxml.py                                 225    160     70      3    24%
.tox/py35/lib/python3.5/site-packages/_pytest/main.py                                     525    184    173     27    57%
.tox/py35/lib/python3.5/site-packages/_pytest/mark.py                                     160     58     70     16    59%
.tox/py35/lib/python3.5/site-packages/_pytest/monkeypatch.py                              125     82     48      7    30%
.tox/py35/lib/python3.5/site-packages/_pytest/nose.py                                      42     17     28      8    50%
.tox/py35/lib/python3.5/site-packages/_pytest/pastebin.py                                  61     45     18      3    24%
.tox/py35/lib/python3.5/site-packages/_pytest/python.py                                   827    418    396     59    43%
.tox/py35/lib/python3.5/site-packages/_pytest/recwarn.py                                  123     97     37      0    16%
.tox/py35/lib/python3.5/site-packages/_pytest/resultlog.py                                 86     67     36      2    17%
.tox/py35/lib/python3.5/site-packages/_pytest/runner.py                                   362    159    120     20    50%
.tox/py35/lib/python3.5/site-packages/_pytest/setuponly.py                                 46     29     28      4    28%
.tox/py35/lib/python3.5/site-packages/_pytest/setupplan.py                                 12      4      4      2    62%
.tox/py35/lib/python3.5/site-packages/_pytest/skipping.py                                 251    160    130     23    30%
.tox/py35/lib/python3.5/site-packages/_pytest/terminal.py                                 442    194    220     49    50%
.tox/py35/lib/python3.5/site-packages/_pytest/tmpdir.py                                    62     39     10      0    32%
.tox/py35/lib/python3.5/site-packages/_pytest/unittest.py                                 150    112     48      4    20%
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/__init__.py                 0      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py                 413    139    144     21    62%
.tox/py35/lib/python3.5/site-packages/pkg_resources/__init__.py                          1506    757    542     70    45%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/__init__.py                     0      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/__about__.py         10      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/__init__.py           3      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/_compat.py           12      1      2      1    86%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/_structures.py       41     17      0      0    59%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/markers.py          122     21     45     12    77%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/requirements.py      72     17     18      1    69%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/specifiers.py       284    157    128     11    36%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/version.py          150     68     88     16    45%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/pyparsing.py                 2483   1314   1062    125    42%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/six.py                        444    209    136     19    49%
.tox/py35/lib/python3.5/site-packages/pkg_resources/extern/__init__.py                     35      5     10      4    80%
.tox/py35/lib/python3.5/site-packages/py/__init__.py                                        5      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/py/_apipkg.py                                       138     46     58     10    62%
.tox/py35/lib/python3.5/site-packages/py/_builtin.py                                      188    145     64      3    18%
.tox/py35/lib/python3.5/site-packages/py/_code/__init__.py                                  0      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/py/_code/code.py                                    523    410    182      2    16%
.tox/py35/lib/python3.5/site-packages/py/_error.py                                         50     35      8      0    26%
.tox/py35/lib/python3.5/site-packages/py/_iniconfig.py                                    123     45     52      9    63%
.tox/py35/lib/python3.5/site-packages/py/_io/__init__.py                                    0      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/py/_io/capture.py                                   266    215    102      1    14%
.tox/py35/lib/python3.5/site-packages/py/_io/terminalwriter.py                            234    150     98     18    33%
.tox/py35/lib/python3.5/site-packages/py/_path/__init__.py                                  0      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/py/_path/common.py                                  244    135     96      8    37%
.tox/py35/lib/python3.5/site-packages/py/_path/local.py                                   628    417    263     34    31%
.tox/py35/lib/python3.5/site-packages/py/_xmlgen.py                                       170    117     50      2    28%
.tox/py35/lib/python3.5/site-packages/py/test.py                                            6      2      2      1    62%
.tox/py35/lib/python3.5/site-packages/pytest.py                                             8      2      2      1    70%
.tox/py35/lib/python3.5/site-packages/pytest_cov/__init__.py                                1      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/pytest_cov/compat.py                                 19      7      4      1    57%
.tox/py35/lib/python3.5/site-packages/pytest_cov/embed.py                                  30     23      8      0    18%
.tox/py35/lib/python3.5/site-packages/pytest_cov/engine.py                                162    136     53      0    12%
.tox/py35/lib/python3.5/site-packages/pytest_cov/plugin.py                                143     99     56      4    24%
reprotest/__init__.py                                                                     236     66     53      3    67%
reprotest/_contextlib.py                                                                   42     11     12      2    69%
reprotest/_shell_ast.py                                                                    89     16     16      1    74%
reprotest/lib/VirtSubproc.py                                                              464    392    140      0    12%
reprotest/lib/__init__.py                                                                   0      0      0      0   100%
reprotest/lib/adt_testbed.py                                                              661    447    287     32    27%
reprotest/lib/adtlog.py                                                                    61     18     16      3    68%
tests/tests.py                                                                             27      6      8      1    69%
-------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                   17817   9644   6795    858    41%

Normally I would run this with --omit .tox/* also to eliminate all the non-reprotest coverage reports, but for comparison with the other reports, I left that out here. Notice that because of the [paths] section in .coveragerc, the coverage measurements for .tox/py35/lib/python3.5/site-packages/reprotest/ are merged with the ones for reprotest/.

With commands = py.test --cov --cov-report=term-missing {posargs} tests/tests.py:

.tox/py35/bin/diffoscope                                                                    6      0      2      1    88%
.tox/py35/lib/python3.5/distutils/__init__.py                                              74     38     26      2    38%
.tox/py35/lib/python3.5/site-packages/_pytest/_argcomplete.py                              33     32     14      1     4%
.tox/py35/lib/python3.5/site-packages/_pytest/_code/code.py                               565    552    188      2     2%
.tox/py35/lib/python3.5/site-packages/_pytest/_code/source.py                             287    280    126      0     2%
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/__init__.py                        69     62     29      2     9%
.tox/py35/lib/python3.5/site-packages/_pytest/assertion/rewrite.py                        557    477    188     13    15%
.tox/py35/lib/python3.5/site-packages/_pytest/cacheprovider.py                            139    117     61     10    18%
.tox/py35/lib/python3.5/site-packages/_pytest/capture.py                                  324    281     98     12    14%
.tox/py35/lib/python3.5/site-packages/_pytest/compat.py                                   113     87     42     10    23%
.tox/py35/lib/python3.5/site-packages/_pytest/config.py                                   837    706    360     22    15%
.tox/py35/lib/python3.5/site-packages/_pytest/debugging.py                                 88     81     18      1     8%
.tox/py35/lib/python3.5/site-packages/_pytest/doctest.py                                  174    171     50      2     2%
.tox/py35/lib/python3.5/site-packages/_pytest/fixtures.py                                 651    309    226     45    53%
.tox/py35/lib/python3.5/site-packages/_pytest/helpconfig.py                                96     87     32      5    11%
.tox/py35/lib/python3.5/site-packages/_pytest/junitxml.py                                 225    223     70      1     1%
.tox/py35/lib/python3.5/site-packages/_pytest/main.py                                     525    318    173     23    38%
.tox/py35/lib/python3.5/site-packages/_pytest/mark.py                                     160     94     70     16    43%
.tox/py35/lib/python3.5/site-packages/_pytest/monkeypatch.py                              125    113     48      3     9%
.tox/py35/lib/python3.5/site-packages/_pytest/nose.py                                      42     28     28      8    34%
.tox/py35/lib/python3.5/site-packages/_pytest/pastebin.py                                  61     59     18      1     4%
.tox/py35/lib/python3.5/site-packages/_pytest/python.py                                   827    554    396     59    32%
.tox/py35/lib/python3.5/site-packages/_pytest/resultlog.py                                 86     84     36      1     2%
.tox/py35/lib/python3.5/site-packages/_pytest/runner.py                                   362    249    120     20    30%
.tox/py35/lib/python3.5/site-packages/_pytest/setuponly.py                                 46     39     28      4    15%
.tox/py35/lib/python3.5/site-packages/_pytest/setupplan.py                                 12     10      4      2    25%
.tox/py35/lib/python3.5/site-packages/_pytest/skipping.py                                 251    212    130     15    14%
.tox/py35/lib/python3.5/site-packages/_pytest/terminal.py                                 442    324    220     30    26%
.tox/py35/lib/python3.5/site-packages/_pytest/tmpdir.py                                    62     55     10      0    10%
.tox/py35/lib/python3.5/site-packages/_pytest/unittest.py                                 150    143     48      4     5%
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py                 413    277    144     15    35%
.tox/py35/lib/python3.5/site-packages/diffoscope/__init__.py                               81     34     24      3    55%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/__init__.py                  132     42     46     10    58%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/ar.py                         27     10      8      0    49%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/binary.py                    215    114     72      9    40%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/bzip2.py                      27      9      0      0    67%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/cbfs.py                       82     33     22      2    55%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/cpio.py                       16      2      0      0    88%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/deb.py                       103     64     44      0    27%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/debian.py                    120    114     40      0     4%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/debian_fallback.py            32     15      6      0    45%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/device.py                     31     17      4      0    40%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/dex.py                        28      9      0      0    68%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/directory.py                 119     78     28      1    29%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/elf.py                       268    167     48      0    33%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/fonts.py                      16      3      0      0    81%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/fsimage.py                    57     34     10      0    34%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/gettext.py                    36     19      8      0    39%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/git.py                        30     15      4      0    44%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/gzip.py                       29     10      0      0    66%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/haskell.py                    51     30     16      2    34%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/icc.py                        14      2      0      0    86%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/image.py                      17      3      0      0    82%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/ipk.py                         6      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/iso9660.py                    43     23      8      0    39%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/java.py                       22      7      2      0    62%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/json.py                       17      7      2      1    58%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/libarchive.py                129     71     64     15    38%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/llvm.py                       15      3      0      0    80%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/macho.py                      53     28      6      0    42%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/mono.py                       14      2      0      0    86%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/pdf.py                        19      4      0      0    79%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/png.py                        19      5      2      0    67%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/ppu.py                        46     29     12      1    31%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/ps.py                         22     10      2      0    50%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/rpm.py                        66     61     14      0     6%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/rpm_fallback.py               13      5      2      0    53%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/rust.py                       37     14      2      1    62%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/sqlite.py                     12      2      0      0    83%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/squashfs.py                  144     81     20      0    38%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/symlink.py                    28     15      4      0    41%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/tar.py                        13      1      0      0    92%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/text.py                       25     15      6      0    32%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/utils.py                     215    108     44      5    43%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/xz.py                         27      9      0      0    67%
.tox/py35/lib/python3.5/site-packages/diffoscope/comparators/zip.py                        90     39      8      0    54%
.tox/py35/lib/python3.5/site-packages/diffoscope/config.py                                 39      0      2      0   100%
.tox/py35/lib/python3.5/site-packages/diffoscope/difference.py                            315    101    119     31    61%
.tox/py35/lib/python3.5/site-packages/diffoscope/main.py                                  134     38     46     15    63%
.tox/py35/lib/python3.5/site-packages/diffoscope/presenters/__init__.py                     0      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/diffoscope/presenters/html.py                       342    298    162      0     9%
.tox/py35/lib/python3.5/site-packages/diffoscope/presenters/icon.py                         1      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/diffoscope/presenters/text.py                        31     15     14      3    47%
.tox/py35/lib/python3.5/site-packages/libarchive/__init__.py                                6      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/libarchive/entry.py                                  72     38      6      0    44%
.tox/py35/lib/python3.5/site-packages/libarchive/exception.py                              10      6      0      0    40%
.tox/py35/lib/python3.5/site-packages/libarchive/extract.py                                49     23      4      0    49%
.tox/py35/lib/python3.5/site-packages/libarchive/ffi.py                                   125     18     20      1    81%
.tox/py35/lib/python3.5/site-packages/libarchive/read.py                                   47     29      2      0    37%
.tox/py35/lib/python3.5/site-packages/libarchive/write.py                                  89     67     12      0    22%
.tox/py35/lib/python3.5/site-packages/magic.py                                            156     43     32     11    67%
.tox/py35/lib/python3.5/site-packages/pkg_resources/__init__.py                          1506    856    542     55    37%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/__init__.py                     0      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/__about__.py         10      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/__init__.py           3      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/_compat.py           12      1      2      1    86%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/_structures.py       41     17      0      0    59%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/markers.py          122     69     45      5    34%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/requirements.py      72     17     18      2    68%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/specifiers.py       284    190    128      4    25%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/packaging/version.py          150     76     88     14    39%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/pyparsing.py                 2483   1354   1062    121    40%
.tox/py35/lib/python3.5/site-packages/pkg_resources/_vendor/six.py                        444    210    136     20    48%
.tox/py35/lib/python3.5/site-packages/pkg_resources/extern/__init__.py                     35      5     10      4    80%
.tox/py35/lib/python3.5/site-packages/py/_apipkg.py                                       138    119     58      3    12%
.tox/py35/lib/python3.5/site-packages/py/_builtin.py                                      188    187     64      0     1%
.tox/py35/lib/python3.5/site-packages/py/_error.py                                         50     47      8      0     5%
.tox/py35/lib/python3.5/site-packages/py/_io/terminalwriter.py                            234    150     98     18    33%
.tox/py35/lib/python3.5/site-packages/py/_path/common.py                                  244    199     96      8    18%
.tox/py35/lib/python3.5/site-packages/py/_path/local.py                                   628    524    263     27    18%
.tox/py35/lib/python3.5/site-packages/py/_xmlgen.py                                       170    168     50      1     1%
.tox/py35/lib/python3.5/site-packages/pytest_cov/compat.py                                 19     15      4      1    22%
.tox/py35/lib/python3.5/site-packages/pytest_cov/engine.py                                162    151     53      2     6%
.tox/py35/lib/python3.5/site-packages/pytest_cov/plugin.py                                143    119     56      9    17%
.tox/py35/lib/python3.5/site-packages/reprotest/__init__.py                               236     66     53      3    67%
.tox/py35/lib/python3.5/site-packages/reprotest/_contextlib.py                             42     11     12      2    69%
.tox/py35/lib/python3.5/site-packages/reprotest/_shell_ast.py                              89     16     16      1    74%
.tox/py35/lib/python3.5/site-packages/reprotest/lib/VirtSubproc.py                        464    392    140      0    12%
.tox/py35/lib/python3.5/site-packages/reprotest/lib/__init__.py                             0      0      0      0   100%
.tox/py35/lib/python3.5/site-packages/reprotest/lib/adt_testbed.py                        661    447    287     32    27%
.tox/py35/lib/python3.5/site-packages/reprotest/lib/adtlog.py                              61     18     16      3    68%
.tox/py35/lib/python3.5/site.py                                                           433    396    215     10     7%
reprotest/__init__.py                                                                     236    156     53      2    30%
reprotest/_contextlib.py                                                                   42     35     12      0    13%
reprotest/_shell_ast.py                                                                    89     26     16      0    60%
reprotest/lib/VirtSubproc.py                                                              464    233    140     33    47%
reprotest/lib/__init__.py                                                                   0      0      0      0   100%
reprotest/lib/adt_testbed.py                                                              661    608    287      0     6%
reprotest/lib/adtlog.py                                                                    61     18     16      3    68%
reprotest/virt/null                                                                        28      2      6      2    88%
tests/tests.py                                                                             27      6      8      1    69%
-------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                   22561  15006   8058    828    30%

Here, the .tox/py35/lib/python3.5/site-packages/reprotest/ measurements are not merged with the ones for reprotest/.

For comparison, when I pass --source=reprotest/ to coverage run, I get this:

reprotest/__init__.py             236     66     53      3    67%
reprotest/__main__.py               3      3      2      0     0%
reprotest/_contextlib.py           42     11     12      2    69%
reprotest/_shell_ast.py            89     16     16      1    74%
reprotest/lib/VirtSubproc.py      464    392    140      0    12%
reprotest/lib/__init__.py           0      0      0      0   100%
reprotest/lib/adt_binaries.py      72     72     26      0     0%
reprotest/lib/adt_run_args.py     188    188     72      0     0%
reprotest/lib/adt_testbed.py      661    447    287     32    27%
reprotest/lib/adtlog.py            61     18     16      3    68%
reprotest/lib/testdesc.py         265    265    110      0     0%
-----------------------------------------------------------------
TOTAL                            2081   1478    734     41    25%

When I pass --cov=reprotest/ instead of --cov to py.test with pytest-cov enabled, I get the same list of files, but coverage reports that it collected no data so the coverage measurements are all zero except for the empty __init__.py file.

[ryanhiebert]

I also had issues with combining files for the purpose of normalizing tox coverage paths. It would run the tests, but the output would always show the paths for the files in the tox virtualenv, not the source file paths I wanted to see.

I wanted it to replace these commands that I've seen commonly, and used myself.

coverage run --parallel-mode -m pytest {posargs}
coverage combine
coverage report

I eventually figured out that this could be done by enabling parallel mode from the coverage configuration (which I like to put in tox.ini).

[coverage:run]
source = mypackage
parallel = True

[coverage:paths]
source =
    src/
    .tox/*/lib/python*/site-packages/

Unfortunately, it still didn't work. It appears that somehow the combine call doesn't work the way that I, and I expect the author that originally implemented the feature, expected it to.

I will be submitting a pull request shortly with the code change that I have found fixes the problem. I'm not clear on whether I'm using the coverage API correctly, but hopefully I can get feedback on it in a code review on that pull request.

[ryanhiebert]

As promised: #142

[ionelmc]

@ceridwen note that --cov=reprotest/ (or --cov=reprotest) is automatically converted to an absolute path. That filter will remove everything if the sources are installed into a virtualenv. You can either do develop installs or use a src-layout for your project (to avoid confusing paths for package names).

@ryanhiebert in my projects it just works, although I have a bit different path configuration:

[paths]
source =
   src/packagename
   */site-packages/packagename

What exactly is wrong with the combine call?

[ryanhiebert]

The problem with the combine call as-is is that it doesn't combine runs from parallel mode. Are you using tox and it "just works"? Note that this isn't for xdist, I'm not using parallel mode for that, but just to normalize the paths.

The differences between our files are a) I only cared to watch for files installed in tox virtualenvs, and b) I don't mention the packagename, mostly because what I'm currently testing is just a module, not a package, and I think that would look weird.

[ionelmc]

Yes yes, not xdist. This is the full config https://github.com/ionelmc/python-nameless/blob/master/.coveragerc and output https://travis-ci.org/ionelmc/python-nameless/jobs/188214771#L223 (paths are src/nameless but code runs from .tox/py35/lib/python3.5/site-packages/nameless).

What am I missing here?

[ryanhiebert]

I'm not sure, that looks basically identical to what I'm trying to do. I'll look at it closer in a little bit. Thank you, @ionelmc .

[ryanhiebert]

@ionelmc : OK, there's two things that you're missing.

First, you're using src as the run.source. While that will work for an editable install, it's not going to work here, because those aren't the files that are actually running. It'll show those file names, but it'll peg the coverage at 0, except for the tests which are the ones that are being run. Instead, you should be using the package name. In your case it's nameless. If you do that, you'll see the same problem I'm having.

Second, it's not combining the files the way it should be. Once the combining is done, all the files left in parallel mode, with filenames like .coverage.Ryans-MacBook-Air.local.77550.764876, should be gone, or at least overwritten in favor of .coverage, which is the file that can be uploaded to codecov.

It's possible that you'd like there to be the option to not combine those parallel files immediately, for instance if you're running multiple tox jobs that will need to be combined with something after all of those jobs. If that's what you prefer, I can add a new flag or something to make that work. Normalizing the paths in that case probably isn't going to work without some more work to allow us to combine the combined files, since the path normalization happens during combining.

However in my case, and I suspect most cases, it's more important to display the coverage the way it's expected than to make files easily combinable, since having it combine between tox envs is going to mean some kind of orchestration outside of tox, and in any case between pytest runs.

---

So that you can see the real code that I'm working with, here's some links. Note that I don't have Travis set up to run these tests yet. I'm still working on other things, I wasn't planning on getting Travis and everything set up quite yet. Also note that all the interesting stuff is in the tox.ini, I've not bothered using .coveragerc or setup.cfg to store the coverage configuration.

I'll just give a link to the tox.ini file, but feel free to browse everything else if you like. The changes are all in the tox.ini file, and the diff from the problem version is given here inline. I removed the .tox directory and any .coverage* files between each run.

The problem version

https://github.com/ryanhiebert/lektor-git/blob/d079bf0/tox.ini

output (truncated, from tox):

---------- coverage: platform darwin, python 2.7.12-final-0 ----------
Name                                                  Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------------------
.tox/py27/lib/python2.7/site-packages/lektor_git.py      21      0      6      0   100%

ls -a .coverage*:

.coverage.Ryans-MacBook-Air.local.80826.387809

The paths aren't normalized, but it is showing that it's at 100% coverage. The file remaining is the parallel version of the file, not the desired .coverage file.

Like nameless

https://github.com/ryanhiebert/lektor-git/blob/8b339c0/tox.ini

diff:

diff --git a/tox.ini b/tox.ini
index fb5451c..6a50f6a 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,7 +12,7 @@ commands =

 [coverage:run]
 branch = True
-source = lektor_git
+source = src
 parallel = True

 [coverage:paths]

output:

---------- coverage: platform darwin, python 2.7.12-final-0 ----------
Name                Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------
src/lektor_git.py      21     21      6      0     0%

ls -a coverage*:

.coverage.Ryans-MacBook-Air.local.81137.720498

This is showing the paths I want, but the coverage is wrong, saying 0% instead of 100%. The file remaining is the parallel version of the file, not the desired .coverage file.

With the patch from #142

https://github.com/ryanhiebert/lektor-git/blob/14a0f36/tox.ini

diff:

diff --git a/tox.ini b/tox.ini
index fb5451c..aaa6e5b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -6,7 +6,7 @@ whitelist_externals = git
 deps =
     pytest
     pytest-mock
-    pytest-cov
+    git+https://github.com/ryanhiebert/pytest-cov.git@062590574c231f68648cbbfc029669843b519de6#egg=pytest-cov
 commands =
     pytest --cov {posargs}

output:

---------- coverage: platform darwin, python 2.7.12-final-0 ----------
Name                Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------
src/lektor_git.py      21      0      6      0   100%

ls -a .coverage*:

.coverage

The paths are normalized the way I want, and the coverage is correctly showing 100%. The file remaining is the desired .coverage file, and there are no parallel versions.

---

Sorry this got so long-winded. I wanted to clarify the problem for you, so I haven't yet looked more into the test failures.

[ryanhiebert]

Oh yeah, now there's easy steps to reproduce the problem:

# Make sure tox is installed
git clone git@github.com:ryanhiebert/lektor-git.git
cd lektor-git
git checkout d079bf0  # git hash in case I push more commits
tox

[ionelmc]

This does the trick:

diff --git a/tox.ini b/tox.ini
index fb5451c..4f56c00 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,10 +12,10 @@ commands =

 [coverage:run]
 branch = True
-source = lektor_git
+source = src
 parallel = True

 [coverage:paths]
 source =
-    src/
-    .tox/*/lib/python*/site-packages/
+    src/lektor_git.py
+    */site-packages/lektor_git.py

You have a point about the stray .coverage.blablabla file - but that looks like a different problem.

[ryanhiebert]

With the patch you provided above, it is still showing incorrect coverage, because it's not measuring the files are are run, it's just measuring the files at the path. But the files at the path aren't run, because they have been installed into tox's virtualenv.

Here's the report output. Notice that the coverage is 0%, not the correct 100%.

---------- coverage: platform darwin, python 2.7.12-final-0 ----------
Name                Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------
src/lektor_git.py      21     21      6      0     0%

The parallel coverage file is just a symptom of the issue, which is that files are not being combined. If they were being combined, then we'd have a proper .coverage file left over instead of, or at least in addition to, the parallel version.

BTW, if you just install to a virtualenv manually instead of using tox, you'll hit this same problem when you run pytest --cov.