Repository navigation
Python Unittest isues #7176
Description
Issue Type: Bug
Synopsis: unexpected behaviours - (1) running a file of tests doesn't run all tests in the file, (2) auto-configuration of testing subdir doesn't seem complete/correct; feature request - (3) nice if could have a 'run tests' button for a single file or selected subset of tests.
My test case is biopython - not small or minimal, but easy to access and a good example of large, collaborative project that I don't get to change the structure of to make things work. I believe I have all the Unittest functionality working, below is what I had to do to make it so. I think with the info below #5252 can be closed?
Tests are in a 'Tests' subdirectory, named test_*.py, based on python unittest. Putting this info into the Testing configuration tool, it set up python.unitTest.cwd to ${workspaceFolder}/Tests and the -s option in python.unittest.args to ./Tests.
Issue: In order to make test discovery work, I needed to change .Tests to ${workspaceFolder}/Tests in python.unittest.args -s option.
Issue: When I run a test_*.py file with the green 'play' arrow, the '?' changes to a green checkbox indicating test passed. However, on opening the file in the outline pane, the indication is that only the first test has been run.
Issue: When I click the 'bug' icon next to the 'play' arrow to debug a unittest on a test that reads from a subdirectory, the test is running in ${workspaceFolder} instead of ${workspaceFolder}/Tests -- even though the run test ('play') succeeded fine. To fix this I created a test entry in launch.json (entry found in some old files, don't know where it came from so new user would need more help here) and added a $cwd entry:
{
"name": "Unit Tests",
"type": "python",
"request": "test",
"cwd": "${workspaceFolder}/Tests"
}
Feature Request: Biopython has 287 files in the Tests subdir, it would be really nice if the 'Run tests' button in the bottom status bar could be configured to e.g. just run the 12 files of tests relevant to the section of code that I am working on.
Despite these issues, I really, really applaud the testing framework in VS Code! I have never written unit tests before (suspect I'm not alone in this), and having the nice, slick framework and buttons all set up makes me want to implement and use them. Trying not to overstate this, but having this all working easy and friendly really seems like it could have a huge impact on the world of coding quality.
Extension version: 2019.8.30787
VS Code version: Code 1.37.1 (f06011ac164ae4dc8e753a3fe7f9549844d15e35, 2019-08-15T16:16:34.800Z)
OS version: Darwin x64 18.7.0
System Info
| Item | Value |
|---|---|
| CPUs | Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (4 x 3200) |
| GPU Status | 2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: enabled oop_rasterization: unavailable_off protected_video_decode: unavailable_off rasterization: unavailable_off skia_deferred_display_list: disabled_off skia_renderer: disabled_off surface_synchronization: enabled_on video_decode: enabled viz_display_compositor: disabled_off webgl: enabled webgl2: enabled |
| Load (avg) | 2, 2, 2 |
| Memory (System) | 32.00GB (9.98GB free) |
| Process Argv | |
| Screen Reader | no |
| VM | 0% |
Activity
karrtikr commentedon Sep 4, 2019
The default value of
cwdis${workspaceFolder}, so it's not surprising. Even when thecwdis set to${workspaceFolder}, you should be able to debug tests. It's just that these particular tests are assuming that they will always be run from${workspaceFolder}/testsdirectory (which is not correct, as tests can be run from anywhere), due to which debugging fails.So it's not a bug, but we can do a few things here out of the box:
cwdby default pointing it to directory set by-sflag, instead of the${workspaceFolder}"request": "test"entry inlaunch.json. We can probably add it in suggestions, so that they can atleast know they can set the current working directory :The second option doesn't seem right, because the user is not supposed to run the
"test"debug configuration. So we could go with option 1 as a feature request.karrtikr commentedon Sep 5, 2019
I created a new issue #7204 to track this, I hope that addresses your concern.
rob-miller commentedon Sep 5, 2019
Thanks, I have upvoted #7204. Sorry I hit the 'close' button
rob-miller commentedon Sep 5, 2019
I was thinking that a 'request: test' entry should be generated in launch.json as needed by test configuration, but did not realise it was not required for the test framework in general. Otherwise it seems like several 'user discovery' steps are still needed to make things work after s/he described the desired situation in the test configuration sequence?
Ideally this debug entry would be auto-selected when debug icon clicked in test explorer, but I can see how this might cause more problems.
rob-miller commentedon Sep 5, 2019
OK, I have done more testing and conclude that the issue is a conflict with the python.testing.cwd entry. There is caching involved to add confusion, so on the Mac I have to close the VS Code window (but not 'quit' from the menu) and re-open at the "Editing evolved" screen where I pick the folder to work on to see the effect of changing python.testing.cwd.
In my case I need "python.testing.cwd": "${workspaceFolder}/Tests" in order for the tests to succeed, as they read files from Tests/subdir. I think setting testing.cwd does not happen in the test configuration process. With this setting, python.testing.unittestArgs "-s ." and "-s ${workspaceFolder}/Tests" work for test discovery, but "-s ./Tests" does not (this now makes sense to me).
Please see if you can reproduce this? I can see that it would be expected behavior but it was confusing.
Not knowing the further ramifications especially for the other testing frameworks, it seems like a better situation here would be for test configuration to set testing.cwd for a testing subdir and leave it out of testing.unittestargs (no -s option at all works for me). This way the user would not have to discover testing.cwd if they need it.
karrtikr commentedon Sep 5, 2019
Yes, it's better not to expose it to users. We add the configuration ourselves internally while sending the debug configurations to debugger.
Will be looking into the other issue shortly.
karrtikr commentedon Sep 6, 2019
If you have both
"python.testing.cwd": "${workspaceFolder}/Tests"and"-s ./Tests", then it won't work because"./Tests"would be resolved to"${workspaceFolder}/Tests/Tests". I think you realise this now.I understand it was confusing but I don't think there's much we can do here.
-soption is definitely needed because it gives the location where to discover tests. It's just for your particular configuration, it didn't work. Also setting"python.testing.cwd"could affect other aspects of testing, so setting it just for your case doesn't seem right.Thanks for reporting these anyways. Closing it now as issues has been addressed.
karrtikr commentedon Sep 6, 2019
You can let me know if there's a better approach here, and we can reconsider doing something about it.
rob-miller commentedon Sep 7, 2019
Yes, I understand the situation.
May I suggest adding some further warning or information about the conflict to the current testing.cwd and testing.unittest text:
Not sure what the best wording is, but somehow something should highlight an issue if testing.cwd is set. Presumably testing.cwd similarly affects the other testing frameworks?
karrtikr commentedon Sep 7, 2019
So basically the message we want to convey is that, if
testing.cwdis set, then.in directory paths resolve to the value of the setting, right? (If you would like this, please open up a new issue on this, marking it as code health). I think it would be better if somehow let user know the directory user is discovering tests in. Maybe in the logs.And yes, it affects other frameworks.