Skip to content

Adding documetation for VSTest Integration #402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 21, 2019
Merged

Adding documetation for VSTest Integration #402

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dbf0c0e
Adding documetation for VSTest integration
vagisha-nidhi May 7, 2019
1249828
Review comments
vagisha-nidhi May 8, 2019
59a7331
Updating Documentation casing
vagisha-nidhi May 8, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions documentation/VSTestIntegration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Coverlet Integration with VSTest

## Motivation
The cross platform solution for Code Coverage generation for .NET Core projects(in a consistent manner).
The asks for Code Coverage support for .NET Core on Linux is the most commented issue on vstest repo:
https://github.com/Microsoft/vstest/issues/981#issuecomment-320311552

## Summary
What would integrating Coverlet with Microsoft Test Platform mean:

1. Coverlet based coverage solution is available as a data collector that does the instrumentation of the necessary modules before test execution and appropriate restore after.
2. Authoring - When new test projects are created (dotnet mstest), references to the data collector package are added by default. This reduces adoption friction for customers.
3. Test execution - Today Coverlet is invoked as a msbuild target and as such only works with dotnet test (and requires a csproj). With the data collector becoming available, coverage can also be collected when tests are run on built binaries (dotnet vstest)

## Proposed Solution

### Scenarios to support
The following table summarizes the support that needs to be added for a seamless code coverage collection for .NET Core on both Linux and Windows platforms:

| Entry point | How will code coverage be enabled? | Syntax |
|-------------|------------------------------------|----------------------------------------------------------------------|
|dotnet test CLI | Through a switch to condition data collection | `dotnet test --collect:"XPlat Code Coverage"` |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this real final name or only a placeholder?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XPlat Code Coverage is the final name at present.

|dotnet vstest CLI | Through a switch to condition data collection | `dotnet vstest --collect:"XPlat Code Coverage"` |

### Coverlet Options Supported with VSTest

#### Default
| Option | Summary |
|-------------|------------------------------------|
|Format | Results format in which coverage output is generated. Default format is cobertura.|
Copy link
Collaborator

@MarcoRossignoli MarcoRossignoli May 7, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we provide default CoverletOutput?
With msbuild we are using MSBuildProjectDirectory https://github.com/tonerdo/coverlet/blob/0956d1e8da6b7265b0b5e4f93af3499c4be99cc8/src/coverlet.msbuild.tasks/coverlet.msbuild.props#L14 but there is some discussion in place #152

I'm in agreement with @eerhardt comment I prefer put it in output directory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will only be getting the coverage output from coverlet.core APIs and the responsibility of sending the file will be with the datacollector.
By default, currently the files are put into the TestResults directory, Where this TestResults directory will be created is configurable via runsettings.


#### Advanced Options (Supported via runsettings)
These are a list of options that are supported by coverlet. These can be specified as datacollector configurations in the runsettings.

| Option | Summary |
|------------- |------------------------------------------------------------------------------------------|
|Format | Coverage output format. These are either cobertura, json, lcov, opencover or teamcity. |
|MergeWith | Combine the output of multiple coverage runs into a single result. |
|Exclude | Exclude from code coverage analysing using filter expressions. |
|ExcludeByFile | Ignore specific source files from code coverage. |
|Include | Explicitly set what to include in code coverage analysis using filter expressions. |
|IncludeDirectory| Explicitly set which directories to include in code coverage analysis. |
|SingleHit | Specifies whether to limit code coverage hit reporting to a single hit for each location.|
|UseSourceLink | Specifies whether to use SourceLink URIs in place of file system paths. |

How to specify these options via runsettings?
```
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat code coverage">
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tech question, test runner search for a class with DataCollectorFriendlyName attribute named XPlat code coverage from output build folder, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It searches for the class in various folders including the output build folder and the nuget folders.
Basically we'll call a target before VSTest that would add the nuget path of the datacollectors to the search directories(it will be picked up directly from the nuget folder)

<Configuration>
<Format>json</Format>
<MergeWith>/custom/path/result.json</MergeWith>
<Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*</Exclude> <!-- [Assembly-Filter]Type-Filter -->
<Include>[coverlet.*]*,[*]Coverlet.Core*</Include> <!-- [Assembly-Filter]Type-Filter -->
<ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
<ExcludeByFile>../dir1/class1.cs,../dir2/*.cs,../dir3/**/*.cs,</ExcludeByFile> <!-- Absolute or relative file paths -->
<IncludeDirectory>../dir1/*,../dir2/,</IncludeDirectory>
<SingleHit>false</SingleHit>
<UseSourceLink>true</UseSourceLink>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
```
This runsettings file can easily be provided using command line option as given :

1. `dotnet test --settings coverletArgs.runsettings`

2. `dotnet vstest --settings coverletArgs.runsettings`


#### Scope of Enhancement
Currently, advanced options are supported via runsettings. Providing support through additional command line arguments in vstest can be taken up separately.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Providing support through additional command line arguments in vstest can be taken up separately.

Will be better user experience...but I think that it's require non trivial internal changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. This will include some internal vstest changes which we can pick up as an enhancement.


## Implementation Details
The proposed solution is implemented with the help of [datacollectors](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md).
1. Outproc Datacollector : The outproc collector would always run in a separate process(datacollector.exe/datacollector.dll) than the process in which tests are being executed(testhost*.exe/testhost.dll). This datacollector would be responsible for calling into coverlet APIs for instrumenting dlls, collecting coverage results and sending the coverage output file back to test platform.
2. Inproc Datacollector : The inproc collector in the testhost process executing the tests. This collector will be needed to remove the dependency on the exit handler to flush the hit files.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see config to plug inproc datacollector...am I missing something?

Copy link
Contributor Author

@vagisha-nidhi vagisha-nidhi May 7, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inproc datacollector will be automatically plugged in with --collect:"XPlat Code Coverage" . The inproc datacollector will lie in the same nuget and get picked from there.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok thank's now it's more clear in/out behaviours of my past tests.


The datacollectors will be bundled as a separate NuGet package, the reference to which will be added by default in the .NET Core test templates, thus making it the default solution for collecting code coverage for .NET core projects.
```
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />
<PackageReference Include="MSTest.TestAdapter" Version="1.3.2" />
<PackageReference Include="MSTest.TestFramework" Version="1.3.2" />
<PackageReference Include="coverlet.collector" Version="1.0.0" />
</ItemGroup>
```