Update: Conversation is moved to here
Hello,
VSCode has native test coverage visualisation implemented. For C++ with all the existing tools/option it is not trivial how to integrate it so I would like to start a discussion about it.
My best idea so far to create an API for TestMate C++ (TM for shorthand) which allows anyone to extend the functionality of this extension with another extension which will interact with the specific coverage framework. Let's call it CE as Coverage Extension.
Proposed API:
import * as vscode from 'vscode';
//////////////////////////////////
/* Guide:
* Be aware that an executable can be run parallel so the coverage information might be generated parallel. Are the sharing some resources?
* Use the token and check cancellation!
* Check `endProcess` for `result` status, maybe only successful run should be handled. Depends on your case and intetions.
* Use `finalise` and/or `onDidDispose` to clean things up / free up memory.
*/
///////////////////////////////////
export interface TestMateTestRun {
/**
* A cancellation token which will be triggered when the test run is
* canceled from the UI.
*/
readonly token: vscode.CancellationToken;
/**
* Adds coverage for a file in the run.
*/
addCoverage(fileCoverage: vscode.FileCoverage): void;
/**
* Probably don't need to use it but just in case...
* Appends raw output from the test runner. On the user's request, the
* output will be displayed in a terminal. ANSI escape sequences,
* such as colors and text styles, are supported. New lines must be given
* as CRLF (`\r\n`) rather than LF (`\n`).
*
* @param output Output text to append.
*/
appendOutput(output: string): void;
/**
* An event fired when the editor is no longer interested in data
* associated with the test run.
*/
onDidDispose: vscode.Event<void>;
}
///
export interface TestMateProcessBuilder {
cwd: string;
cmd: string;
args: string[];
env: Record<string, string | undefined>;
}
export interface TestMateTestRunHandler {
/**
* The same executable can be run parallel so the coverage information might be generated parallel.
* Or they might sharing some resources exclusively? In that case: `false`
* Can be dynamic:
* ```
* get allowExecutableConcurrentInvocations() { return vscode.workspace.getConfiguration(configSection).get<boolean>('allowExecutableConcurrentInvocations', false); }
* ```
* Limitation applies for `TestMateTestRunProfile` instance, so other profile/Coverage tool can be run parallel with this.
*/
readonly allowExecutableConcurrentInvocations?: boolean;
/**
* During this callback, one can do the global init part.
* Use `testRun.token` !!!
* @param progress Same as in case of {@linkcode vscode.window.withProgress}
*/
init?: (progress: vscode.Progress<{ message?: string; increment?: number }>) => void | Promise<void>;
/**
* Called before the executable's process is spawned.
* Use `testRun.token` !!!
* @param builder if `mapTestRunProcessBuilder` is defined the the its result value
*/
beginProcess?: (builder: TestMateProcessBuilder) => void | Promise<void>;
/**
* Called after the executable's process is spawned.
* Use `testRun.token` !!!
* @param builder if {@linkcode mapTestRunProcessBuilder} is defined the the its result value
*/
endProcess?: (
builder: TestMateProcessBuilder,
result: 'OK' | 'CancelledByUser' | 'TimeoutByUser' | 'Errored',
) => void | Promise<void>;
/**
* Will be called after all the tests (TestRun) are finished. Can do some cleanup and/or summing up the results.
* Use `testRun.addCoverage` to add/update/overwrite coverage info. https://code.visualstudio.com/api/extension-guides/testing#test-coverage
* Use `testRun.token` !!!
*
* Note: Do not dispose your detailed coverage, `loadDetailedCoverage` still can be called.
* Can use WeakMap<vscode.FileCoverage, MyCoverageDetails> or subclassing. See vscode api docs
* @param progress Same as in case of {@linkcode vscode.window.withProgress}
*/
finalise?: (progress: vscode.Progress<{ message?: string; increment?: number }>) => void | Promise<void>;
/**
* If you need to change something for the call of the exec, return with the modifed.
* IMPORTANT: Whatever is the result of this mapping the process's output should not change to remain parsable!!
* IMPORTANT: ONLY test run will apply the mapped value, discover and test listing WONT.
* Example:
* ```
* return {
* ...builder,
* cmd: 'coverage.exe',
* args: ['--exec', builder.cmd, '--exec_args', ...builder.args],
* env: { ...builder.env, OVERRIDE_THIS: 'new value' },
* }
* ```
*
* @param builder Executable would be called with this normally
* @returns Executable will be called with this.
*/
mapTestRunProcessBuilder?: (
builder: TestMateProcessBuilder,
) => TestMateProcessBuilder | Promise<TestMateProcessBuilder>;
}
export interface TestMateTestRunProfile {
/**
* Will be used to create coverage profile. Example: "gcov (by ...)"
*/
readonly label: string;
/**
* You probably want to use `vscode.TestRunProfileKind.Coverage`
*/
readonly kind: vscode.TestRunProfileKind;
/*
* Profile test tag. If specified, only the tests having the same tag can be used with this profile.
* Best practice to give an option to your users to set this tag and they can also set `testMate.cpp.test.advancedExecutables[].testTags` as well.
*/
readonly tag?: vscode.TestTag;
/**
* TestMate will call this when user initiates a test run
*/
createTestRunHandler(testRun: TestMateTestRun, workspaceFolder: vscode.WorkspaceFolder): TestMateTestRunHandler;
/**
* See: https://code.visualstudio.com/api/extension-guides/testing#test-coverage
* @param fileCoverage same as were added by testRun.addCoverage(...)
* @param token Must be used!
*/
loadDetailedCoverage?: (
testRun: TestMateTestRun,
fileCoverage: vscode.FileCoverage,
token: vscode.CancellationToken,
) => Promise<vscode.FileCoverageDetail[]>;
/**
* If this method is present, a configuration gear will be present in the
* UI, and this method will be invoked when it's clicked. When called,
* you can take other editor actions, such as showing a quick pick or
* opening a configuration file.
*/
configureHandler?: () => void;
dispose(): void;
}
export interface TestMateAPI {
/**
* Call this to register your Coverage Adapter. Can be called multiple times if has multiple profiles. (See vscode api for more info)
*/
registerTestRunProfile(adapter: TestMateTestRunProfile): void;
}Example usage:
Update: Conversation is moved to here
Hello,
VSCode has native test coverage visualisation implemented. For C++ with all the existing tools/option it is not trivial how to integrate it so I would like to start a discussion about it.
My best idea so far to create an API for TestMate C++ (TM for shorthand) which allows anyone to extend the functionality of this extension with another extension which will interact with the specific coverage framework. Let's call it CE as Coverage Extension.
Proposed API:
Example usage:
src/llvm-cov.ts,src/gcov.ts