Pytest discovery fails if something is printed to stderr

[1st]

Can I get any help with pytest discovery failure?

> /usr/local/bin/python ~/.vscode-server/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear
cwd: .
[ERROR 2022-1-3 23:36:9.283]: Error discovering pytest tests:
 [r [Error]: Graceful shutdown completed in 0.000 seconds.

	at ChildProcess.<anonymous> (/root/.vscode-server/extensions/ms-python.python-2022.0.1786462952/out/client/extension.js:32:39235)
	at Object.onceWrapper (events.js:422:26)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)]

Tried many different options with "python.testing.cwd" and "python.testing.pytestArgs" but without luck.

I used unit tests in the past and they work fine in VS Code.

Also I use VS Code with "Remote Container" and work inside the Docker container and run tests inside it.

Hope that you can solve this issue soon, because with pytest it doesn't work for me. And I have no clear vision why. Thanks.

[1st]

Hi @kimadeline

Please let me know if I can provide any useful details that may help to identify the issue. Thanks

[1st]

What I may say, in my current project it's two apps in one codebase. Depending on the env variable (that is set in the .env file) - it may be run in one or another way.

When I run the tests, I do it like this APP_ENV=test APP_NAME=backend pytest backend and it runs only the tests for the part of the project. If I need to run other part of tests - I do APP_ENV=test APP_NAME=frontend pytest frontend

I did not find how make the test discovery that is used by VS Code to specify env variables.

All what I need from VS Code discovery - just to show me a list of all tests I have in the codebase. And also to have a "run" icon near each test function.

Currently I do run tests manually. I have defined in my launch.json next snippet. I change it to run individual test, if I do not need to run all tests in the file:

        {
            "name": "Run current test",
            "type": "python",
            "request": "launch",
            "module": "pytest",
            "env": {
                "APP_ENV": "test",
                "APP_NAME": "backend",
            },
            "justMyCode": false,
            "args": [
                "--no-cov",
                // Test current file
                "${file}",
                // Filter tests by name
                "-k test_some_feature_name",
            ],
            "console": "integratedTerminal",
        },

It works to me fine. Only thing that I do often - opening the launch.json file and change the -k flag each time when I need to run other test. And it's annoying.

Maybe it's a way to make Test Discovery works fine with pytest for such a project?

[calebho]

Getting a similar error, however it seems like it's because the environment location is messed up

> conda run -n onedevex-dev --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2022.0.1786462952/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .
cwd: .
[ERROR 2022-1-4 19:17:3.378]: Error discovering pytest tests:
 [r [Error]: 
EnvironmentLocationNotFound: Not a conda environment: /private/home/calebh/miniconda3/envs/onedevex-dev/envs/onedevex-dev


	at ChildProcess.<anonymous> (/private/home/calebh/.vscode-server/extensions/ms-python.python-2022.0.1786462952/out/client/extension.js:32:39235)
	at Object.onceWrapper (events.js:422:26)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)

[brianoflondon]

Perhaps I'm getting something similar which seems to stem from the discovery process trying to run my module itself with bad command line options. I'm really new to setting up a new project here so very unsure if this is my problem or similar to what others have reported here.

This bit puzzles me, why is discovery trying to run podping-hivewatcher (my project) with pytest command line options.

usage: hivewatcher [options]
hivewatcher: error: unrecognized arguments: discover pytest -- --rootdir /Users/gbishko/Documents/Python-iMac/PodcastIndex/podping-hivewatcher -s --cache-clear tests

Running pytest on the command line works fine.

> ~/Library/Caches/pypoetry/virtualenvs/podping-hivewatcher-YKYAFVom-py3.8/bin/python ~/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
cwd: .
[ERROR 2022-1-5 14:29:48.180]: Error discovering pytest tests:
 [r [Error]: ============================= test session starts ==============================
platform darwin -- Python 3.8.9, pytest-5.4.3, py-1.11.0, pluggy-0.13.1
rootdir: /Users/gbishko/Documents/Python-iMac/PodcastIndex/podping-hivewatcher
usage: hivewatcher [options]
hivewatcher: error: unrecognized arguments: discover pytest -- --rootdir /Users/gbishko/Documents/Python-iMac/PodcastIndex/podping-hivewatcher -s --cache-clear tests
collected 0 items / 1 error

==================================== ERRORS ====================================
________ ERROR collecting tests/integration/test_podping_hivewatcher.py ________
tests/integration/test_podping_hivewatcher.py:2: in <module>
    from podping_hivewatcher.hivewatcher import allowed_op_id, get_allowed_accounts
src/podping_hivewatcher/hivewatcher.py:10: in <module>
    from podping_hivewatcher.config import Config
src/podping_hivewatcher/config/__init__.py:164: in <module>
    args = my_parser.parse_args()
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/argparse.py:1771: in parse_args
    self.error(msg % ' '.join(argv))
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/argparse.py:2521: in error
    self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/argparse.py:2508: in exit
    _sys.exit(status)
E   SystemExit: 2
=========================== short test summary info ============================
ERROR tests/integration/test_podping_hivewatcher.py - SystemExit: 2
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.46s ===============================

Traceback (most recent call last):
  File "/Users/gbishko/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/Users/gbishko/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/Users/gbishko/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

	at ChildProcess.<anonymous> (/Users/gbishko/.vscode/extensions/ms-python.python-2022.0.1786462952/out/client/extension.js:32:39235)
	at Object.onceWrapper (events.js:422:26)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)]

[ianhelle]

I'm getting a very similar issue - pytest discovery not working in vscode while pytest from cmd line works fine.
From the code Python output I see:

> f:\anaconda\condabin\conda.bat run -p F:\anaconda\envs\msticpy --no-capture-output python ~\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\get_output_via_markers.py ~\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests

Not sure if something in conda is able to interpret ~\.vscode\extens... as %userprofile%\.vscode\extens... (I'm running on Windows where "~" doesn't resolve to anything).

If I try to copy that command (and replace the "~" with userprofile path), it fails as follows

(I've added -v flags to conda run to try to get more info):

(msticpy) E:\src\microsoft\msticpy>f:\anaconda\condabin\conda.bat run -v -v -v -p F:\anaconda\envs\msticpy --no-capture-output python %userprofile%\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\get_output_via_markers.py %userprofile%\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests 
DEBUG conda.gateways.logging:set_verbosity(231): verbosity set to 3
DEBUG conda.gateways.subprocess:subprocess_call(71): executing>> C:\WINDOWS\system32\cmd.exe /d /c F:\anaconda\envs\msticpy\.tmpzij4q0j9.bat
>>>PYTHON-EXEC-OUTPUT
usage: C:\Users\Ian\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py [-h] {discover} ...
C:\Users\Ian\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py: error: argument cmd: invalid choice: 'C:\\Users\\Ian\\.vscode\\extensions\\ms-python.python-2022.1.1779589815-dev\\pythonFiles\\testing_tools\\run_adapter.py' (choose from 'discover')
TRACE conda.gateways.subprocess:subprocess_call(108): $ C:\WINDOWS\system32\cmd.exe /d /c F:\anaconda\envs\msticpy\.tmpzij4q0j9.bat
==> cwd: E:\src\microsoft\msticpy <==
==> exit code: 2 <==
==> stdout <==
None
==> stderr <==
None

ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['python', 'C:\\Users\\Ian\\.vscode\\extensions\\ms-python.python-2022.1.1779589815-dev\\pythonFiles\\get_output_via_markers.py', 'C:\\Users\\Ian\\.vscode\\extensions\\ms-python.python-2022.1.1779589815-dev\\pythonFiles\\testing_tools\\run_adapter.py', 'discover', 'pytest', '--', '--rootdir', '.', '-s', '--cache-clear', 'tests']' command failed.  (See above for error)
TRACE conda.gateways.disk.delete:rm_rf(165): rm_rf F:\anaconda\envs\msticpy\.tmpzij4q0j9.bat

I also tried removing the get_output_via_markers.py component of the command - still no success. But I noticed that
it seemed to be trying to execute pytest from the conda base environment rather than the current conda env.

(msticpy) E:\src\microsoft\msticpy>f:\anaconda\condabin\conda.bat run -v -v -v -p F:\anaconda\envs\msticpy --no-capture-output python %userprofile%\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
DEBUG conda.gateways.logging:set_verbosity(231): verbosity set to 3
DEBUG conda.gateways.subprocess:subprocess_call(71): executing>> C:\WINDOWS\system32\cmd.exe /d /c F:\anaconda\envs\msticpy\.tmp39rluiwl.bat
equivalent command: f:\anaconda\python.exe -m pytest --collect-only --rootdir . -s --cache-clear tests
TRACE conda.gateways.subprocess:subprocess_call(108): $ C:\WINDOWS\system32\cmd.exe /d /c F:\anaconda\envs\msticpy\.tmp39rluiwl.bat
==> cwd: E:\src\microsoft\msticpy <==
==> exit code: 120 <==
==> stdout <==
None
==> stderr <==
None

ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['python', 'C:\\Users\\Ian\\.vscode\\extensions\\ms-python.python-2022.1.1779589815-dev\\pythonFiles\\testing_tools\\run_adapter.py', 'discover', 'pytest', '--', '--rootdir', '.', '-s', '--cache-clear', 'tests']' command failed.  (See above for error)
TRACE conda.gateways.disk.delete:rm_rf(165): rm_rf F:\anaconda\envs\msticpy\.tmp39rluiwl.bat

This is the line I'm referring to - although not sure if this is a anything to do with the problem - it should be f:\anaconda\envs\msticpy\python.exe

equivalent command: f:\anaconda\python.exe -m pytest --collect-only --rootdir . -s --cache-clear tests

The result is the same with or without the -v flags.

[kimadeline]

Hi everyone, a couple of things here:

• Issues with conda are most likely related to Debugging does not work with a conda environment #18436, since we recently changed how we run tools when in a conda environment. We hope to make a hotfix release today or tomorrow, please try it out whenever it comes out, and if it still doesn't work please open a separate issue;
• When creating a conda environment, make sure you create it with the python=3.XXX argument, or the extension won't recognize it as a conda environment;
• We currently do not support setting environment variables in the test runner, we have an open feature request for it: Support setting environmental variables for the test runner #17171. Please upvote it to show your interest and help us prioritize it! A workaround for now would be to specify your environment variables in a .env file.

@1st and @brianoflondon, please provide a sample repo that I can use to try and replicate your issues.