Support for "related code" in tests

[connor4312]

@connor4312 Is there any consideration to how I would present one or more tests as being related to a code item like a function?

For instance, in Pester you can tag tests with anything. If I allowed a special tag that let you say a test was assigned to a certain function named Invoke-MyFunction, I'd want to expose a "play" button right next to the Invoke-MyFunction definition to run all tests "tagged" to that function. In this case though I don't want it to be tracked with a separate ID, I want it to invoke those individual tests as part of the tag "rollup" which might be scattered across different files. Also tricky is how you would present a failure to the user in this case.

Ideally this would work with AutoRun as well whenever I change the function.

At any rate this is definitely a "nice to have" as opposed to a "must have" and could come at a later iteration, just a thought that would make testing really simple for people and not have to lose the context of where they are directly working on the application. I was inspired by your markdown test adapter example.

With the existing implementation it could just be offered under a "Function Test Rollup" header or something and aggregate the results together somehow, but when run it wouldn't update the "referenced" tests necessarily (though I guess I could do this within the context of the extension) so its fragmented and not ideal.

Originally posted by @JustinGrote in #107467 (comment)

[firelizzard18]

This is applicable to Go. go test recognizes three types of test functions: TestXxx, BenchmarkXxx, and ExampleXxx. Go has an established naming convention for associating examples with what they are an example of. If you follow this naming convention, pkg.go.dev/go doc/etc will associate each example with the thing it is an example of. This naming convention could be used to attach examples (which are a kind of test) to functions/etc.

As far as I know, that naming convention isn't used for test and benchmark functions, but that's probably because those aren't included in documentation. I think it would be reasonable to use the same convention to attach tests and benchmarks to functions/etc.

I am the author of the Go Test Explorer extension, and I may be submitting PRs to vscode-go to implement the testing API.

[connor4312]

@jasonrudolph recently did some work on a signficant-other extension that implements something like this. Our friends on the Java team (@jdneo) released similar functionality in November, and @brettcannon wants this in Python too.

How would the API for this look like? Would it be something along the lines of related?: vscode.Location[] on the TestItem? Something more or less advanced?

[JustinGrote]

How would the API for this look like? Would it be something along the lines of related?: vscode.Location[] on the TestItem? Something more or less advanced?

I don't see why it would need to be more complicated than this, since the extension can also keep updating the location if the code/tests move.

An open question is how to handle "one to many" situations in both directions.

[jdneo]

An open question is how to handle "one to many" situations in both directions.

In Java, we leverage the reference-view to deal with the "one to many" case.

[firelizzard18]

How would the API for this look like? Would it be something along the lines of related?: vscode.Location[] on the TestItem? Something more or less advanced?

@connor4312 I think Location[] is a good first step. I do think it would be valuable to have something like relatedSymbols?: vscode.DocumentSymbol[]. The most useful features that come to mind are:

1. Auto-run a test whenever the user changes the referenced location. This can be done with Location.
2. Allow the user to see what symbols are linked to a test, similar to (or incorporated into) find references. This would be much more straight forward if the test item can explicitly link to DocumentSymbols.

[connor4312]

(2) is interesting, how would you envision the UI/UX for that?

[JustinGrote]

Linking to symbols would be useful, since you wouldn't have to constantly update the location for add-on commands, but it could always be a dropdown menu on right-click for the test (maybe even a tree expansion)

[connor4312]

The other simple alternative is a pair of functions such as provideImplementationRanges(testItem: vscode.TestItem): ProviderResult<vscode.Range[]> / provideAssociatedTests(range: vscode.Range): ProviderResult<vscode.TestItem>, which VS Code can call on demand when the user asks to go between tests and implementation. This is likely more efficient for implementors.