When running dynamically generated tests for module, only the first one runs

[alexvy86]

Environment data

• VS Code version: 1.31.1
• Extension version (available under the Extensions sidebar): 2019.2.5433
• OS and version: Windows 10 Pro
• Python version (& distribution if applicable, e.g. Anaconda): 3.7.2
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A (but also experiencing it with pipenv)
• Relevant/affected Python packages and their versions: unittest

Expected behaviour

All dynamically generated test (unittest) methods within a module/file should run when clicking the "Run test" button for that module/file in test explorer, or when using the context menu and selecting "Run current unit test file".

Note that this is already the case when clicking the "Run test" button for the class inside the module/file, in test explorer.

Actual behaviour

When clicking the "Run test" button for the module/file in the test explorer, or when selecting "Run current unit test file" from the context menu, only the first dynamically generated test method runs.

As mentioned above, when clicking the "Run test" button for the class inside the module/file, in the test explorer, then all the dynamically generated tests run fine.

Steps to reproduce:

See the screen capture below with a reproduction. The overall idea is to create a (unittest) test class and generate its test methods dynamically with setattr, then try to run the resulting set of tests. In the screen capture I dynamically generate two tests. I first run the tests for the class (through test explorer), and both generated tests run fine. Then I run them for the file (through test explorer) and only the first one runs. Finally I run them with the context menu and "Run current unit test file" and again only the first one runs.

The project in the screen capture can be found in this branch.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Microsoft Python Language Server version 0.1.80.0
Initializing for C:\Python37\python.exe

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

The following set of messages show up for any of the 3 test cases described above (run module in test explorer, run class in test explorer, run tests in current file from context menu).

[Extension Host] Python Extension: Cached data exists ActivatedEnvironmentVariables, f:\python-unittest-repro    console.ts:134
[Extension Host] Python Extension: getActivatedEnvironmentVariables, Class name = b, Arg 1: <Uri:f:\python-unittest-repro>, Arg 2: undefined    console.ts:134
[Extension Host] Python Extension: Cached data exists getEnvironmentVariables, f:\python-unittest-repro    console.ts:134
[Extension Host] Python Extension: Error: read ECONNRESET    console.ts:134 
  | t.log | @ | console.ts:134
  | e.$logExtensionHostMessage | @ | mainThreadConsole.ts:39
  | t._doInvokeHandler | @ | rpcProtocol.ts:390
  | t._invokeHandler | @ | rpcProtocol.ts:375
  | t._receiveRequest | @ | rpcProtocol.ts:295
  | t._receiveOneMessage | @ | rpcProtocol.ts:225
  | (anonymous) | @ | rpcProtocol.ts:100
  | e.fire | @ | event.ts:567
  | s | @ | ipc.net.ts:295
  | (anonymous) | @ | ipc.net.ts:302
  | e.fire | @ | event.ts:567
  | a | @ | ipc.net.ts:103
  | _socketDataListener | @ | ipc.net.ts:132
  | emit | @ | events.js:182
  | addChunk | @ | _stream_readable.js:279
  | readableAddChunk | @ | _stream_readable.js:264
  | Readable.push | @ | _stream_readable.js:219
  | onread | @ | net.js:636

[Ronkiro]

Similar to #4567

[DonJayamanne]

No, its not the same as #4567

[ericsnowcurrently]

Thanks for letting us know about this, @alexvy86. I've been able to reproduce the problem using the file you provided:

import unittest

class TestClass(unittest.TestCase):
    pass


def make_dynamic_test_function(text: str):
    def test_function(self):
        self.assertEqual(text, "Correct")

    return test_function

test_parameters = [
    "Incorrect",
    "Correct",
]

for item in test_parameters:
    setattr(TestClass,
        f"test_text__{item}",
        make_dynamic_test_function(item))

We'll work on fixing this.

[alexvy86]

I saw that today's release of the extension included a fix for #4567 so I decided to give this another try, and it works now! So maybe they were indeed the same thing :). Closing the issue.