Python Unittest isues

[rob-miller]

Issue Type: Bug

Synopsis: unexpected behaviours - (1) running a file of tests doesn't run all tests in the file, (2) auto-configuration of testing subdir doesn't seem complete/correct; feature request - (3) nice if could have a 'run tests' button for a single file or selected subset of tests.

My test case is biopython - not small or minimal, but easy to access and a good example of large, collaborative project that I don't get to change the structure of to make things work. I believe I have all the Unittest functionality working, below is what I had to do to make it so. I think with the info below #5252 can be closed?

Tests are in a 'Tests' subdirectory, named test_*.py, based on python unittest. Putting this info into the Testing configuration tool, it set up python.unitTest.cwd to ${workspaceFolder}/Tests and the -s option in python.unittest.args to ./Tests.

Issue: In order to make test discovery work, I needed to change .Tests to ${workspaceFolder}/Tests in python.unittest.args -s option.

Issue: When I run a test_*.py file with the green 'play' arrow, the '?' changes to a green checkbox indicating test passed. However, on opening the file in the outline pane, the indication is that only the first test has been run.

Issue: When I click the 'bug' icon next to the 'play' arrow to debug a unittest on a test that reads from a subdirectory, the test is running in ${workspaceFolder} instead of ${workspaceFolder}/Tests -- even though the run test ('play') succeeded fine. To fix this I created a test entry in launch.json (entry found in some old files, don't know where it came from so new user would need more help here) and added a $cwd entry:

   {
        "name": "Unit Tests",
        "type": "python",
        "request": "test",
        "cwd": "${workspaceFolder}/Tests"
    } 

Feature Request: Biopython has 287 files in the Tests subdir, it would be really nice if the 'Run tests' button in the bottom status bar could be configured to e.g. just run the 12 files of tests relevant to the section of code that I am working on.

Despite these issues, I really, really applaud the testing framework in VS Code! I have never written unit tests before (suspect I'm not alone in this), and having the nice, slick framework and buttons all set up makes me want to implement and use them. Trying not to overstate this, but having this all working easy and friendly really seems like it could have a huge impact on the world of coding quality.

Extension version: 2019.8.30787
VS Code version: Code 1.37.1 (f06011ac164ae4dc8e753a3fe7f9549844d15e35, 2019-08-15T16:16:34.800Z)
OS version: Darwin x64 18.7.0

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (4 x 3200)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: enabled oop_rasterization: unavailable_off protected_video_decode: unavailable_off rasterization: unavailable_off skia_deferred_display_list: disabled_off skia_renderer: disabled_off surface_synchronization: enabled_on video_decode: enabled viz_display_compositor: disabled_off webgl: enabled webgl2: enabled
Load (avg)	2, 2, 2
Memory (System)	32.00GB (9.98GB free)
Process Argv
Screen Reader	no
VM	0%

[karrtikr]

Hi @rob-miller 👋

Issue: In order to make test discovery work, I needed to change .Tests to ${workspaceFolder}/Tests in python.unittest.args -s option.

It should be ./Tests, not .Tests, please try that. This doesn't seem like an issue.

Issue: When I run a test_*.py file with the green 'play' arrow, the '?' changes to a green checkbox indicating test passed. However, on opening the file in the outline pane, the indication is that only the first test has been run.

This is the expected behavior right now. The logic is,
If any test fails, the parent should be failure, otherwise pass if all tests that ran passed, and not run if no tests ran.

So a node is '?' only if all the nodes below it are '?'. If you wish to know more details about a test node, you can just hover over any test item to see if it contains any tests that did not run.

Feature Request: Biopython has 287 files in the Tests subdir, it would be really nice if the 'Run tests' button in the bottom status bar could be configured to e.g. just run the 12 files of tests relevant to the section of code that I am working on.

It could be useful, could please open up a new issue on this feature request, so that we can make a decision on whether we want this.

[karrtikr]

Btw are the test nodes with '?' being skipped intentionally?(using unittest.skip decorator or something) Try running them individually.

[karrtikr]

Issue: When I click the 'bug' icon next to the 'play' arrow to debug a unittest on a test that reads from a subdirectory, the test is running in ${workspaceFolder} instead of ${workspaceFolder}/Tests -- even though the run test ('play') succeeded fine. To fix this I created a test entry in launch.json (entry found in some old files, don't know where it came from so new user would need more help here) and added a $cwd entry:

I will follow up later regarding this issue.

[rob-miller]

Cheers @karrtikr,

Issue: In order to make test discovery work, I needed to change .Tests to ${workspaceFolder}/Tests in python.unittest.args -s option.

It should be ./Tests, not .Tests, please try that. This doesn't seem like an issue.

Sorry, typo in my bug report - it was script-generated as ./Tests and I had to change to ${workspaceFolder}/Tests. When left as ./Tests, test discovery did not find any tests.

Issue: When I run a test_*.py file with the green 'play' arrow, the '?' changes to a green checkbox indicating test passed. However, on opening the file in the outline pane, the indication is that only the first test has been run.

This is the expected behavior right now. The logic is,
[edit]
Btw are the test nodes with '?' being skipped intentionally?(using unittest.skip decorator or something) Try running them individually.

No, and this is the issue I am reporting. Referring to the screenshot, when I run (green arrow) test_PDB_parse_pdb_header.py, I get the result shown (only the test_parse_header_line appears to have run, and if I had not opened the selected line I would think that all four sub-tests ran and passed). If I run from the ParseReal line, all four sub-tests run and succeed.

I will open the feature request point separately as you suggest.

[karrtikr]

No, and this is the issue I am reporting. Referring to the screenshot, when I run (green arrow) test_PDB_parse_pdb_header.py, I get the result shown (only the test_parse_header_line appears to have run, and if I had not opened the selected line I would think that all four sub-tests ran and passed). If I run from the ParseReal line, all four sub-tests run and succeed.

I am able to confirm this, thanks.

[thornhillcody]

I've run into this same problem. It really should be treated as a bug, not a request for a new feature. When running a module of tests, it doesn't make sense that only the first test case should run. I've included a zip of a very small project for you to reproduce. In the Test window, just click the run button on the module as shown:

pyunittestproblem.zip

[karrtikr]

@thornhillcody Like I commented earlier, I am able to confirm this as a bug. This is not expected. Thanks for giving a minimal sample btw, it really helps!

[karrtikr]

No, and this is the issue I am reporting. Referring to the screenshot, when I run (green arrow) test_PDB_parse_pdb_header.py, I get the result shown (only the test_parse_header_line appears to have run, and if I had not opened the selected line I would think that all four sub-tests ran and passed). If I run from the ParseReal line, all four sub-tests run and succeed.

@rob-miller @thornhillcody This issue is a duplicate of #4567. Can you guys please upvote that issue, it helps us prioritize it.