-
Notifications
You must be signed in to change notification settings - Fork 214
Running pytest-cov in parallel #416
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
As I read a few issues, it looks like many people are relying on a Imagine one have a [tox]
envlist = py37, py38
[testenv]
deps = -rrequirements.txt
commands = pytest --cov-report term-missing --cov {envsitepackagesdir}/foo/ and some Python code like: if sys.version_info >= (3, 8):
...
else:
... I don't think we want two distinct coverage files, one whining about the first branch not being tested and the other whining about the other branch not being tested. I think we need a single Using my proposed
gives 3 output files, I'm trying with this "reproducer" https://mdk.fr/x/pytest-cov-reproducer.tar.bz2 Haven't found a way to tell tox "run this once all parallel jobs are done", which would anyay be tox side, not pytest-cov side. So I think we can only document all of this and let user choose what they want according to what they need. |
Hi, I am a pytest-cov user.
This would solve my problem exactly -- I am not using tox, and can afford making a call after-the-fact to combine the results. If I can disable the automatic combining via flag/config, it will make pytest-cov more composable with other paradigms (like MPI). |
So in theory there could be that option, that you can use and get the old broken behavior. But the old broken behavior still read other coverage files (for subprocess support). So then you'd ask how about an option to disable that too right? But then you'd basically make pytest-cov do almost nothing, and you might as well just use I would argue that running tests in parallel with coverage on is a fringe usecase - make up your mind and chose either speed or coverage. |
I solved my issue of running |
The fact coverage run -m pytest to get coverage results instead of pytest and having the I'm uninstalling
|
I just switched from pytest-cov back to solo coverage for the reason above. |
Uh oh!
There was an error while loading. Please reload this page.
Summary
People are using
tox -p auto
ortox -p all
more and more since it exists, (and some are simply using&
in shell scripts).But it's failing with
pytest-cov
(nedbat/coveragepy#883, #356, #237, #217).This is because pytest-cov uses a
coverage combine
step which tries to combine all.coverage.*
files, mixing files from all of the parallels runs. As some are incomplete, this often yield to sqlite errors, but it also sometime just mix the data in strange ways.A clean fix is to specify a specific coverage file name for each run, so the combine step will search for files with this specific name, avoiding mixing the files.
This can easily be done, for example, in
.tox.ini
by using:It make
coverage combine
search for.coverage.py37.*
for example.I see two strategies:
Either pytest-cov picks a unique coverage file name per run or pytest-cov documents that when used in parallel one should specify a coverage file name to disambiguate the runs.
Do you have a preference?
The text was updated successfully, but these errors were encountered: