Stderr output causes test discovery to show as failed

[Cnoor0171]

Issue Type: Bug

Behaviour

Expected vs. Actual

This is duplicate of #7095, but that issue was closed because the author didn't provide steps to reproduce. The bug is still there.

Expected: the extension to be able to detect tests when pytest test discovery succeeds, even if there is output on stderr.

Actual: If there is leftover output on stderr, the extension fails to discover tests. Note that the print needs to happen during test discovery, NOT inside the test. Also note that the discovery only fails if content is written to stderr or stdout after pytest thinks it's done reporting the test collection results. For that to happen we need to spin up a separate thread that waits to print something. In my particular case, the culprit was a library that spawns a background thread for log processing that was logging a warning. Running pytest --collect-only exits cleanly, but the extension is not able to process the results.

The extension logs show:

> ./.venv/bin/python ~/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .
cwd: .
[ERROR 2022-1-19 1:15:57.156]: Error discovering pytest tests:
 [r [Error]: something something something
	at ChildProcess.<anonymous> (/home/cnoor0171/.vscode-server/extensions/ms-python.python-2022.0.1814523869/out/client/extension.js:32:39235)
	at Object.onceWrapper (events.js:422:26)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)]

Steps to reproduce:

1. In an empty workspace, install pytest in a venv
2. Create a file test_something.py with content

from threading import Thread
from time import sleep
import sys

def threaded_function():
    sleep(3)
    sys.stderr.write("something something something")

thread = Thread(target=threaded_function)
thread.start()

def test_passing():
    assert 1 == 1

3. Try to run pytest discovery on workspace

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.9.5
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Pylance

User Settings

defaultLS: {"defaultLSType":"Pylance"}

downloadLanguageServer: true

envFile: "<placeholder>"

venvPath: "<placeholder>"

venvFolders: "<placeholder>"

condaPath: "<placeholder>"

pipenvPath: "<placeholder>"

poetryPath: "<placeholder>"

languageServer: "Pylance"

linting
• enabled: true
• cwd: "<placeholder>"
• Flake8Args: "<placeholder>"
• flake8Enabled: false
• flake8Path: "<placeholder>"
• lintOnSave: true
• banditArgs: "<placeholder>"
• banditEnabled: false
• banditPath: "<placeholder>"
• mypyArgs: "<placeholder>"
• mypyEnabled: false
• mypyPath: "<placeholder>"
• pycodestyleArgs: "<placeholder>"
• pycodestyleEnabled: false
• pycodestylePath: "<placeholder>"
• prospectorArgs: "<placeholder>"
• prospectorEnabled: false
• prospectorPath: "<placeholder>"
• pydocstyleArgs: "<placeholder>"
• pydocstyleEnabled: false
• pydocstylePath: "<placeholder>"
• pylamaArgs: "<placeholder>"
• pylamaEnabled: false
• pylamaPath: "<placeholder>"
• pylintArgs: "<placeholder>"
• pylintPath: "<placeholder>"

sortImports
• args: "<placeholder>"
• path: "<placeholder>"

formatting
• autopep8Args: "<placeholder>"
• autopep8Path: "<placeholder>"
• provider: "autopep8"
• blackArgs: "<placeholder>"
• blackPath: "<placeholder>"
• yapfArgs: "<placeholder>"
• yapfPath: "<placeholder>"

testing
• cwd: "<placeholder>"
• debugPort: 3000
• nosetestArgs: "<placeholder>"
• nosetestsEnabled: undefined
• nosetestPath: "<placeholder>"
• promptToConfigure: true
• pytestArgs: "<placeholder>"
• pytestEnabled: true
• pytestPath: "<placeholder>"
• unittestArgs: "<placeholder>"
• unittestEnabled: false
• autoTestDiscoverOnSaveEnabled: true

terminal
• activateEnvironment: true
• executeInFileDir: "<placeholder>"
• launchArgs: "<placeholder>"

experiments
• enabled: true
• optInto: []
• optOutFrom: []

insidersChannel: "off"

tensorBoard
• logDirectory: "<placeholder>"

Extension version: 2022.0.1814523869
VS Code version: Code 1.64.2 (f80445acd5a3dadef24aa209168452a3d97cc326, 2022-02-09T22:02:28.252Z)
OS version: Windows_NT x64 10.0.19044
Restricted Mode: No
Remote OS version: Linux x64 5.10.16.3-microsoft-standard-WSL2
Remote OS version: Linux x64 5.10.16.3-microsoft-standard-WSL2

System Info

Item	Value
CPUs	AMD Ryzen 5 2600 Six-Core Processor (12 x 3394)
GPU Status	2d_canvas: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: enabled opengl: enabled_on rasterization: enabled skia_renderer: enabled_on video_decode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.93GB (6.67GB free)
Process Argv	--file-uri=vscode-remote://wsl+Ubuntu-20.04/mnt/d/.de1/Code/workspace.code-workspace --remote=wsl+Ubuntu-20.04 --crash-reporter-id 2459b23f-72eb-4ec8-a6b4-b166f7e6ac7d
Screen Reader	no
VM	0%

Item	Value
Remote	WSL: Ubuntu-20.04
OS	Linux x64 5.10.16.3-microsoft-standard-WSL2
CPUs	AMD Ryzen 5 2600 Six-Core Processor (12 x 3393)
Memory (System)	12.43GB (10.35GB free)
VM	0%

Item	Value
Remote	WSL: Ubuntu-20.04
OS	Linux x64 5.10.16.3-microsoft-standard-WSL2
CPUs	AMD Ryzen 5 2600 Six-Core Processor (12 x 3393)
Memory (System)	12.43GB (10.34GB free)
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyl392:30425749
pythontb:30283811
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30438340
pythondataviewer:30285071
vscod805:30301674
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30433507
vsclayoutctrc:30437038
vsrem710:30416614
vscop841cf:30438916
dsvsc009:30440023
vsbas813:30436447
vscscmwlcmc:30438804
helix:30440343

[kimadeline]

Hi @Cnoor0171, thank you for reaching out.

As you experienced, right now we read test output from stdout/stderr, and this is very brittle. We are updating this mechanism as part of #6594 and #17242.

Secondly, the throwOnStdErr option is currently set to true when running test discovery, which we shouldn't do.

[eleanorjboyd]

still exists on rewrite- will need another PR to fix.

[eleanorjboyd]

tried this on the rewrite on insiders and it no longer shows the test as failed on discovery. I was also able to run the test as follows and it works passes.

def test_passing():
    sys.stderr.write("something something something")
    assert 1 == 1

please comment if I tested it incorrectly or if you are still seeing the issue.