Debugger doesn't stop at breakpoints with pytest if pytest-cov is used

[lmazuel]

Environment data

VS Code version: 1.19.3
Python Extension version: 2018.1.0
Python Version: 3.6.2
OS and version: Windows 10 latest update

Actual behavior

My initial setup has this setup.cfg file:

[tool:pytest]
addopts = --cov=mymodulename

This adds automatically a "--cov" to every "pytest" call, but prevents VSCode to stop at breakpoints. If I remove the file (or at least the --cov), I get back my breakpoints stop.
I changed my setup to put that in my Travis file instead, but I feel like this should work :/

Expected behavior

I should be able to have this file (and coverage) and debug at the same time.

Steps to reproduce:

• Install pytest and pytest-cov
• Create a setup.cfg as describe before
• Create even a simple test with a breakpoint inside

[lmazuel]

Note that the test is executed correctly. Even the "run test"/"debug test" button on each test executes correctly this individual test.

[DonJayamanne]

@lmazuel apologies for the delay.
Looks like its not possible to debug the code with coverage enabled (http://pytest-cov.readthedocs.io/en/latest/debuggers.html).
I guess we'll have to disable 'coverage' on the fly when debugging (using the flag "--no-cov").

[DonJayamanne]

Hmm, looks like we're not the only ones having trouble with debugging + pytest + coverage (https://www.jetbrains.com/help/pycharm/run-debug-configuration-py-test.html)

I've tried passing in the flag --no-cov, but that causes pytest to fall over of pytest-cov isnt installed.

I guess we could try to determine if Pytest-cov is installed or not.

[DonJayamanne]

@lmazuel
Unfortunately we're not going to be able to resolve this easily, hence we're closing this.
My suggestion is to modify the pytest args in the settings.json to manually pass in the --no-cov flag to enable debugging.
I understand this isn't the best solution, as you'll be modifying the settings on and off.

[lmazuel]

That's ok :). And I like the fact you open a documentation issue ;)

[justfalter]

@DonJayamanne I don't really like having to edit my workspace settings any time I want to switch back and forth between generating coverage and debugging.
What about the idea of specifying an extra set of arguments that would be appended to python.unitTest.pyTestArgs when the "debug unit test" feature is used? It'd give users a place to put "--no-cov", if they happen to be using coverage. Maybe call it python.unitTest.pyTestDebugExtraArgs?

[MaximeWeyl]

Thanks for the --no-cov trick !
I have to edit my configuration anytime I want to switch between coverage and debug, so I hope someone develops the enhancement proposed by @justfalter

Since I don't know how to develop VS Code plugins, I will use the trick for now :-)

[zevisert]

I suspect unittest and nosetest also skip breakpoints when coverage is enabled, judging by their documentation,[1] [2], since they also use sys.settrace(). The vscode-python docs only explictly mention coverage with pytest debugging [3].

Following @justfalter's suggestion, I'd like to suggest another scheme for settings.json:

First, I think it's better off that the .unitTest becomes .testing:

- python.unitTest.*
+ python.testing.*

Second, I'd suggest instead implementing new settings namespaces fpr debug mode. These would be python.testing.{py,unit,nose}testDebugArgs, which if specified, is used instead of python.testing.{py,unit,nose}testArgs when running in debug mode, and falling back to ...{py,unit,nose}testArgs when not debugging.

Ref:
1: Coverage.py for unittest
2: Nosetest also using Coverage.py
3. VSCode only notes skipping breakpoints with pytest

[zevisert]

@limonkufu Put simply, there's no fix. Think about it this way:

1. Coverage is enabled and tests are run in debug mode
2. A breakpoint is hit and the program is paused
3. You run a command in the debug terminal that calls parts of your code

There's no straightforward (and maintainable) way that the coverage tracker can know if the parts of your code that the debug terminal invoked should count towards coverage or not. The coverage frameworks disable breakpoints so that they can get accurate coverage reporting. It's for this reason that I suggested a new settings.json namespace above so that we can specify different arguments for test debugging (i.e. with coverage disabled)

[limonkufu]

@zevisert thanks for explaining it clearly, I didn't understand it before, if this is beginner friendly to this repo, I would like work on in. I'll investigate a little bit thanks

…

On Thu, 18 Mar 2021 at 03:50, Zev Isert ***@***.***> wrote: @limonkufu <https://github.com/Limonkufu> Put simply, there's no fix. Think about it this way: 1. Coverage is enabled and tests are run in debug mode 2. A breakpoint is hit and the program is paused 3. You run a command in the debug terminal that calls parts of your code There's no straightforward (and maintainable) way that the coverage tracker can know if the parts of your code that the debug terminal invoked should count towards coverage or not. The coverage frameworks disable breakpoints so that they can get accurate coverage reporting. It's for this reason that I suggested a new settings.json namespace above so that we can specify different arguments for test debugging (i.e. with coverage disabled) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#693 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2LFHAPTYINQU53NEOMDRTTEFZ75ANCNFSM4EO7DNXA> .

[dferrante]

for those coming to this thread, i have a solution that may work for some people. it involves moving your pytest args into pytest.ini within your project, setting up two tasks to sed and un-sed the coverage args, and adding these tasks to your launch.json.

sample pytest.ini (this is for a django project):

[pytest]
addopts = --cov-config=.coveragerc --cov-report html --cov-report xml --cov=. --nomigrations --reuse-db

tasks in tasks.json that use sed to overwrite the pytest.ini:

        {
            "label": "coverage-off",
            "type": "shell",
            "command": "sed -i 's/addopts = --cov-config=.coveragerc --cov-report html --cov-report xml --cov=./addopts =/' pytest.ini",
            "problemMatcher": []
        },
        {
            "label": "coverage-on",
            "type": "shell",
            "command": "sed -i 's/addopts =/addopts = --cov-config=.coveragerc --cov-report html --cov-report xml --cov=./' pytest.ini",
            "problemMatcher": []
        }

launch.json for debug tests:

        {
            "name": "Debug Tests",
            "type": "python",
            "preLaunchTask": "coverage-off",
            "justMyCode": false,
            "postDebugTask": "coverage-on",
            "request": "test",
            "console": "integratedTerminal",
        }

slightly annoying solution as it pops open a task window in the terminal, but hey, it works. switch in your own pytest args, or if its only coverage, you can use sed to just comment out the line.

[SigmaX]

I don't suppose there is a way to throw an error when this happens? Preferably a descriptive one, but any error would be helpful.

The docs are appreciated, but since the debugger failure is silent, I wasn't sure if ignoring breakpoints in tests was the intended behavior or not (much less whether coverage should be something I Google docs on).

[AlexanderWells-diamond]

I use an alternative to @dferrante 's solution above, which seems to be seamless inside the VSCode IDE for pytest tests:

In setup.cfg I have:

[tool:pytest]
addopts =  --cov=<module name> --cov-report term --cov-report xml:cov.xml  <other arguments here>

And then in launch.json I define a new task:

 {
    "name": "Debug Unit Test",
    "type": "python",
    "request": "launch",
    "justMyCode": false,
    "program": "${file}",
    "purpose": ["debug-test"],
    "console": "integratedTerminal",
    "env": {
        "PYTEST_ADDOPTS": "--no-cov"
    },
}

When you press the "Debug Test" button in the testing panel or by right-clicking on a test itself, VSCode executes this task which just turns off all coverage for that run. No need to switch between configurations or altering config arguments on the fly.

Note this may only work for fairly new versions of VSCode - I'm on 1.60.

[LightCC]

This is an excellent case for better integration of run/debug configurations. You can currently already create these in the launch.json, but I can't figure out how to get pytest to use the different configurations.

The "Run and Debug" window dropdown will look like this:

With the launch.json file like this:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Py: Current File",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "cwd": "${workspaceRoot}",
      "console": "integratedTerminal",
      "debugOptions": [
        "RedirectOutput"
      ]
    },
    {
      "name": "Py: MD2MAT",
      // OTHER LINES REMOVED
    },
    {
      "name": "PyTest: Normal",
      "args": [],
      // OTHER LINES REMOVED
    },
    {
      "name": "PyTest: Coverage",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "cwd": "${workspaceRoot}",
      "console": "integratedTerminal",
      "justMyCode": true,
      "args": ["--cov" ],
      "debugOptions": [
        "RedirectOutput"
      ]
    }
  ]
}

My pytest.ini is now:

[pytest]
minversion = 6.0
python_functions = test*
testpaths = ./
addopts = --cov-branch --cov-report html --cov-report xml:coverage.xml --cov-report term

My change was to remove the --cov (or --cov=xxx) argument from the pytest.ini addops line, then created a launch.json configuration for "Pytest: Normal" with no extra arguments, and a "Pytest: Coverage" configuration with the --cov=xxx.

This setup works fine at the command line - i.e. just running pytest will run without coverage and pytest --cov will run with coverage. But switching to the "Pytest Coverage" configuration in the Run/Debug window doesn't work to get the test explorer or "Run / Debug" codelens functions to run pytest with the --cov option.

Am I missing something in the configuration/setup? Which configuration does the extension run pytest under, or does it setup/run a custom or hardcoded configuration?

We should be able to setup special configurations for pytest that we can swap between that alter the command-line arguments. This would allow for more than just this coverage fix - I have other use cases for special command line args needed for including/excluding tests for different kinds of test runs (i.e. sanity vs. full test suite) that need the same kind of extra configuration setup.

[pjanowski]

There's no straightforward (and maintainable) way that the coverage tracker can know if the parts of your code that the debug terminal invoked should count towards coverage or not. The coverage frameworks disable breakpoints so that they can get accurate coverage reporting. It's for this reason that I suggested a new settings.json namespace above so that we can specify different arguments for test debugging (i.e. with coverage disabled)

Wouldn't the cleanest solution be for VSCode to automatically disable code coverage when a test is run in debug mode? User side you would see no code coverage when you run tests in debug but code coverage appear fine if you run without debug. That would be pretty clean and painless for the two scenarios (rather than having to fuss with run settings and env variables).

[MaximeWeyl]

seams like a good idea to me

[ericchansen]

Just commenting that this is still a pain in the bum. I want to believe that @AlexanderWells-diamond's answer works because it's the only suggestion with a reasonable level of difficulty/complexity, but it doesn't for me. My only working solution is to uninstall pytest-cov and modify my setup.cfg when debugging.

Here's my workspace configuration.

{
    "folders": [
        ...,
        {
            "path": "/home/ehansen/repos/blah"
        },
        {
            "path": "/home/ehansen/repos/whatever"
        },
        ...
    ],
    "launch": {
        "compounds": [],
        "configurations": [
            {
                "console": "integratedTerminal",
                "cwd": "${fileWorkspaceFolder}",
                "env": {"PYTEST_ADDOPTS": "--no-cov"},
                "justMyCode": false,
                "name": "Python: Debug",
                "purpose": [
                    "debug-test"
                ],
                "request": "launch",
                "type": "python"
            },
            {
                "console": "integratedTerminal",
                "env": {"PYTEST_ADDOPTS": "--no-cov"},
                "name": "Python: Current File",
                "program": "${file}",
                "request": "launch",
                "type": "python"
            },
            {
                "console": "integratedTerminal",
                "env": {"PYTEST_ADDOPTS": "--no-cov"},
                "cwd": "${fileWorkspaceFolder}",
                "name": "Python: Current File in Current Workspace",
                "program": "${file}",
                "request": "launch",
                "type": "python"
            },
            {
                "args": [
                    "so",
                    "many",
                    "args"
                ],
                "console": "integratedTerminal",
                "name": "blah: Specific to this repo",
                "program": "${workspaceFolder:blah}/src/blah/ihatevscoderightnow.py",
                "request": "launch",
                "type": "python"
            },
            {
                "args": [
                    "wow",
                    "args"
                ],
                "console": "integratedTerminal",
                "name": "whatever: Specific to repo",
                "program": "${workspaceFolder:whatever}/src/whatever/thisisabigbummer.py",
                "request": "launch",
                "type": "python"
            },
            ...
        ],
        "version": "0.2.0"
    },
    "settings": {
        "files.trimTrailingWhitespace": true,
        "git.enableCommitSigning": true,
        "python.pythonPath": "/home/ehansen/virtualenvs/thisshouldjustwork/bin/python3.6",
        "python.testing.autoTestDiscoverOnSaveEnabled": true,
        "python.testing.pytestEnabled": true,
        "python.testing.unittestArgs": [
            "-v",
            "-s",
            ".",
            "-p",
            "*test*.py",
            "-f"
        ],
        "python.testing.unittestEnabled": true,
        "terminal.integrated.profiles.linux": {
            "bash": {
                "args": [
                    "-l"
                ],
                "path": "/bin/bash"
            }
        }
    }
}

Here's setup.cfg.

[metadata]
name = blah
...
author = Eric Hansen
...

[options]
zip_safe = False
packages = find_namespace:
include_package_data = True
package_dir =
    = src
install_requires =
    importlib-metadata; python_version<"3.8"

[options.packages.find]
where = src
exclude =
    tests

[options.extras_require]
testing =
    setuptools
    pytest
    pytest-cov

[tool:pytest]
addopts =
    --cov aiplatform --cov-report term-missing
    --verbose
norecursedirs =
    dist
    build
    .tox
testpaths = tests

[sgbaird]

For some reason it stops at manually set breakpoints for me, but not if an error is raised. So, a silly workaround (at least for me) has been to let it run until it errors out, track down the relevant line, and set a breakpoint on it. Obviously not a great solution for long-running tests, or when the same function gets called multiple times before the error occurs.

[KevinEvansSrAeries]

Its something in vscode. If you right-click your test in "testing" then select Debug from the menu it works fine and you can hit your breakpoints in "testing". The debug "button" on the test does not allow the process to stop on a breakpoint.

[skilkis]

@ericchansen seems to me that you are specifying launch configuration in settings.json. I ran into the same issue as you. What fixed it is realizing that the docs specify that customized configurations are parsed from the launch.json file in the .vscode folder of the current workspace. I hope moving your configuration out of settings.json into launch.json will fix this issue for you as well!

On a side note, it would be nice if these launch settings could be parsed from settings.json since it is cumbersome to add a launch.json into every Python project individually.

[sgbaird]

@skilkis could you include an example of a launch.json that resolves the issue for you?

[skilkis]

@sgbaird here is the launch.json that resolved the issue for me. I've hidden it from view to not clutter the debug configuration menu 😊

{
  "version": "0.2.0",
  "configurations": [
    {
      // Disable cov to allow breakpoints when launched from VS Code Python
      "name": "Python: Debug Tests",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "purpose": ["debug-test"],
      "console": "internalConsole",
      "justMyCode": false,
      "presentation": {
        "hidden": true
      },
      "env": {
        "PYTEST_ADDOPTS": "--no-cov"
      }
    }
  ]
}

EDIT: Original fix was provided by @AlexanderWells-diamond in the answer above

[ericchansen]

@skilkis I use multi-root workspaces. launch.json isn't part of that workflow. The launch section of the workspace file should be treated the same as a launch.json (and if it's not being treated the same, that's a bug).

I can confirm that debugging in 1.68.0-insider doesn't work with this workspace (see below). You can see that all of the relevant settings in the launch section of this multi-root workspace are the same as @skilkis's settings.

{
    "folders": [
        {
            "path": "../../../repos/repo-a"
        },
        {
            "path": "../../../repos/repo-b"
        }
    ],
    "launch": {
        "compounds": [],
        "configurations": [
            {
                "console": "integratedTerminal",
                "env": {
                    "PYTEST_ADDOPTS": "--no-cov"
                },
                "justMyCode": false,
                "name": "Python: Debug",
                "purpose": [
                    "debug-test"
                ],
                "request": "launch",
                "type": "python"
            },
            {
                "console": "integratedTerminal",
                "env": {
                    "PYTEST_ADDOPTS": "--no-cov"
                },
                "name": "Python: Current File",
                "program": "${file}",
                "request": "launch",
                "type": "python"
            },
            {
                "console": "integratedTerminal",
                "cwd": "${fileWorkspaceFolder}",
                "env": {
                    "PYTEST_ADDOPTS": "--no-cov"
                },
                "name": "Python: Current File in Current Workspace",
                "program": "${file}",
                "request": "launch",
                "type": "python"
            }
        ],
        "version": "0.2.0"
    },
    "settings": {
        "debug.internalConsoleOptions": "openOnFirstSessionStart",
        "files.trimTrailingWhitespace": true,
        "python.defaultInterpreterPath": "~/virtualenvs/dev/bin/python3",
        "python.testing.autoTestDiscoverOnSaveEnabled": true,
        "python.testing.pytestEnabled": true,
        "python.testing.unittestArgs": [
            "-v",
            "-s",
            ".",
            "-p",
            "*test*.py",
            "-f"
        ],
        "python.testing.unittestEnabled": true,
        "terminal.integrated.profiles.linux": {
            "bash": {
                "args": [
                    "-l"
                ],
                "path": "/bin/bash"
            }
        }
    }
}

The only way to make the debugger work is to remove addopts = --cov repo-a ... from your pyproject.toml, setup.cfg, etc.

If it works in a launch.json as @skilkis claims, but not in a workspace file, that's a bug.

[skilkis]

@ericchansen I've created a minimal reproduction of your configuration and it also does not work for me. The code I used to test it is located at skilkis/vscode_python_pytest_multi_root_breakpoint_issue. Would you mind testing the launch.json file I linked above in a non multi-root workspace? This so that we can confirm that it works fine on your system.

After this I believe we can make a feature request to have support for settings.json and multi-root workspaces!

[ShannonTully]

@skilkis I tested this with launch.json vs. the "launch" setting in settings.json.

# settings.json
// REDACTED
    "launch": {
        "configurations": [
            {
                "name": "Python: Global Debug Tests",
                "type": "python",
                "request": "launch",
                "program": "${file}",
                "purpose": ["debug-test"],
                "console": "integratedTerminal",
                "justMyCode": true,
                "env": {"PYTEST_ADDOPTS": "--no-cov"}
            }
        ],
        "compounds": []
    },
// REDACTED

vs.

# launch.json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Debug Tests",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "purpose": ["debug-test"],
            "console": "integratedTerminal",
            "justMyCode": true,
            "env": {"PYTEST_ADDOPTS": "--no-cov"}
        }
    ]
}

Both options show up in the "Run and Debug" sidebar and work as expected if selected. But if I comment out the contents of launch.json or delete the file, the "Python: Global Debug Tests" option will still be there but will not apply the env variable meaning the debugger will not work correctly. This is definitely a bug with the "launch" option vs. launch.json behavior.

You can recreate this with any tests.

[edmundsj]

This is actually worse than it seems at first blush. Not only will running with coverage enabled fail to respect breakpoints in PyCharm, but I've also found it causes manually inserted breakpoint() lines to stop at locations that are unpredictable and not the specified breakpoint. Not even close. This was with using the arguments of --cov and --cov-report=term.

[samueloph]

This is what I use under settings.json to workaround this globally for the "Testing" feature (not sure about "Run and Debug"):

"python.testing.pytestArgs": [
    // Coverage is not supported by vscode:
    // https://github.com/Microsoft/vscode-python/issues/693
    // Note that this will make pytest fail if pytest-cov is not installed,
    // if that's the case, then this option needs to be be removed (overrides
    // can be set at a workspace level, it's up to you to decide what's the
    // best approach). You might also prefer to only set this option
    // per-workspace (wherever coverage is used).
    "--no-cov"
],

As mentioned in the comment in the setting, I make sure I have pytest-cov installed in my environment, even if that project is not making use of it. If I don't want or can't install pytest-cov, I can set overrides per workspace.

I don't use "Run and Debug" a lot (but I do use a lot of "Testing") and don't remember having an issue with this there so I don't know if I already had the workaround in place or I just didn't hit it.

PS.: In my settings I also make use of -s, but that's unrelated to this issue.

[cwebster-99]

Closing as we do not plan to implement this work.