diff --git a/Documentation/GlobalTool.md b/Documentation/GlobalTool.md new file mode 100644 index 000000000..8974407fd --- /dev/null +++ b/Documentation/GlobalTool.md @@ -0,0 +1,206 @@ +# Coverlet as a Global Tool + +To see a list of options, run: + +```bash +coverlet --help +``` + +The current options are (output of `coverlet --help`): + +```bash +Cross platform .NET Core code coverage tool 1.0.0.0 + +Usage: coverlet [arguments] [options] + +Arguments: + Path to the test assembly. + +Options: + -h|--help Show help information + -v|--version Show version information + -t|--target Path to the test runner application. + -a|--targetargs Arguments to be passed to the test runner. + -o|--output Output of the generated coverage report + -v|--verbosity Sets the verbosity level of the command. Allowed values are quiet, minimal, normal, detailed. + -f|--format Format of the generated coverage report. + --threshold Exits with error if the coverage % is below value. + --threshold-type Coverage type to apply the threshold to. + --threshold-stat Coverage statistic used to enforce the threshold value. + --exclude Filter expressions to exclude specific modules and types. + --include Filter expressions to include specific modules and types. + --include-directory Include directories containing additional assemblies to be instrumented. + --exclude-by-file Glob patterns specifying source files to exclude. + --exclude-by-attribute Attributes to exclude from code coverage. + --include-test-assembly Specifies whether to report code coverage of the test assembly. + --single-hit Specifies whether to limit code coverage hit reporting to a single hit for each location. + --merge-with Path to existing coverage result to merge. + --use-source-link Specifies whether to use SourceLink URIs in place of file system paths. +``` + +## Code Coverage + +The `coverlet` tool is invoked by specifying the path to the assembly that contains the unit tests. You also need to specify the test runner and the arguments to pass to the test runner using the `--target` and `--targetargs` options respectively. The invocation of the test runner with the supplied arguments **must not** involve a recompilation of the unit test assembly or no coverage data will be generated. + +The following example shows how to use the familiar `dotnet test` toolchain: + +```bash +coverlet /path/to/test-assembly.dll --target "dotnet" --targetargs "test /path/to/test-project --no-build" +``` + +After the above command is run, a `coverage.json` file containing the results will be generated in the directory the `coverlet` command was run. A summary of the results will also be displayed in the terminal. + +_Note: The `--no-build` flag is specified so that the `/path/to/test-assembly.dll` isn't rebuilt_ + +## Coverage Output + +Coverlet can generate coverage results in multiple formats, which is specified using the `--format` or `-f` options. For example, the following command emits coverage results in the `opencover` format instead of `json`: + +```bash +coverlet --target --targetargs --format opencover +``` + +Supported Formats: + +* json (default) +* lcov +* opencover +* cobertura +* teamcity + +The `--format` option can be specified multiple times to output multiple formats in a single run: + +```bash +coverlet --target --targetargs --format opencover --format lcov +``` + +By default, Coverlet will output the coverage results file(s) in the current working directory. The `--output` or `-o` options can be used to override this behaviour. + +```bash +coverlet --target --targetargs --output "/custom/path/result.json" +``` + +The above command will write the results to the supplied path, if no file extension is specified it'll use the standard extension of the selected output format. To specify a directory instead, simply append a `/` to the end of the value. + +```bash +coverlet --target --targetargs --output "/custom/directory/" -f json -f lcov +``` + +### TeamCity Output + +Coverlet can output basic code coverage statistics using [TeamCity service messages](https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages). + +```bash +coverlet --target --targetargs --output teamcity +``` + +The currently supported [TeamCity statistics](https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages) are: + +| TeamCity Statistic Key | Description | +| :--- | :--- | +| CodeCoverageL | Line-level code coverage | +| CodeCoverageR | Branch-level code coverage | +| CodeCoverageM | Method-level code coverage | +| CodeCoverageAbsLTotal | The total number of lines | +| CodeCoverageAbsLCovered | The number of covered lines | +| CodeCoverageAbsRTotal | The total number of branches | +| CodeCoverageAbsRCovered | The number of covered branches | +| CodeCoverageAbsMTotal | The total number of methods | +| CodeCoverageAbsMCovered | The number of covered methods | + +## Merging Results + +With Coverlet you can combine the output of multiple coverage runs into a single result. + +```bash +coverlet --target --targetargs --merge-with "/path/to/result.json" --format opencover +``` + +The value given to `--merge-with` **must** be a path to Coverlet's own json result format. + +## Threshold + +Coverlet allows you to specify a coverage threshold below which it returns a non-zero exit code. This allows you to enforce a minimum coverage percent on all changes to your project. + +```bash +coverlet --target --targetargs --threshold 80 +``` + +The above command will automatically fail the build if the line, branch or method coverage of _any_ of the instrumented modules falls below 80%. You can specify what type of coverage to apply the threshold value to using the `--threshold-type` option. For example to apply the threshold check to only **line** coverage: + +```bash +coverlet --target --targetargs --threshold 80 --threshold-type line +``` + +You can specify the `--threshold-type` option multiple times. Valid values include `line`, `branch` and `method`. + +```bash +coverlet --target --targetargs --threshold 80 --threshold-type line --threshold-type method +``` + +By default, Coverlet will validate the threshold value against the coverage result of each module. The `--threshold-stat` option allows you to change this behaviour and can have any of the following values: + +* Minimum (Default): Ensures the coverage result of each module isn't less than the threshold +* Total: Ensures the total combined coverage result of all modules isn't less than the threshold +* Average: Ensures the average coverage result of all modules isn't less than the threshold + +The following command will compare the threshold value with the overall total coverage of all modules: + +```bash +coverlet --target --targetargs --threshold 80 --threshold-type line --threshold-stat total +``` + +## Excluding From Coverage + +### Attributes + +You can ignore a method or an entire class from code coverage by creating and applying the `ExcludeFromCodeCoverage` attribute present in the `System.Diagnostics.CodeAnalysis` namespace. + +You can also ignore additional attributes by using the `ExcludeByAttribute` property (short name or full name supported): + +```bash +coverlet --target --targetargs --exclude-by-attribute "Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute" +``` + +### Source Files + +You can also ignore specific source files from code coverage using the `--exclude-by-file` option + - Can be specified multiple times + - Use absolute or relative paths (relative to the project directory) + - Use file path or directory path with globbing (e.g `dir1/*.cs`) + +```bash +coverlet --target --targetargs --exclude-by-file "../dir1/class1.cs" +``` + +### Filters + +Coverlet gives the ability to have fine grained control over what gets excluded using "filter expressions". + +Syntax: `--exclude '[Assembly-Filter]Type-Filter'` + +Wildcards +- `*` => matches zero or more characters +- `?` => the prefixed character is optional + +Examples + - `--exclude "[*]*"` => Excludes all types in all assemblies (nothing is instrumented) + - `--exclude "[coverlet.*]Coverlet.Core.Coverage"` => Excludes the Coverage class in the `Coverlet.Core` namespace belonging to any assembly that matches `coverlet.*` (e.g `coverlet.core`) + - `--exclude "[*]Coverlet.Core.Instrumentation.*"` => Excludes all types belonging to `Coverlet.Core.Instrumentation` namespace in any assembly + - `--exclude "[coverlet.*.tests?]*"` => Excludes all types in any assembly starting with `coverlet.` and ending with `.test` or `.tests` (the `?` makes the `s` optional) + - `--exclude "[coverlet.*]*" --exclude "[*]Coverlet.Core*"` => Excludes assemblies matching `coverlet.*` and excludes all types belonging to the `Coverlet.Core` namespace in any assembly + +```bash +coverlet --target --targetargs --exclude "[coverlet.*]Coverlet.Core.Coverage" +``` + +Coverlet goes a step in the other direction by also letting you explicitly set what can be included using the `--include` option. + +Examples + - `--include "[*]*"` => Includes all types in all assemblies (everything is instrumented) + - `--include "[coverlet.*]Coverlet.Core.Coverage"` => Includes the Coverage class in the `Coverlet.Core` namespace belonging to any assembly that matches `coverlet.*` (e.g `coverlet.core`) + - `--include "[coverlet.*.tests?]*"` => Includes all types in any assembly starting with `coverlet.` and ending with `.test` or `.tests` (the `?` makes the `s` optional) + +Both `--exclude` and `--include` options can be used together but `--exclude` takes precedence. You can specify the `--exclude` and `--include` options multiple times to allow for multiple filter expressions. + +You can also include coverage of the test assembly itself by specifying the `--include-test-assembly` flag. diff --git a/Documentation/MSBuildIntegration.md b/Documentation/MSBuildIntegration.md new file mode 100644 index 000000000..230ef09e4 --- /dev/null +++ b/Documentation/MSBuildIntegration.md @@ -0,0 +1,172 @@ +# Coverlet Integration with MSBuild + +In this mode, Coverlet doesn't require any additional setup other than including the NuGet package in the unit test project. It integrates with the `dotnet test` infrastructure built into the .NET Core CLI and when enabled, will automatically generate coverage results after tests are run. + +If a property takes multiple comma-separated values please note that [you will have to add escaped quotes around the string](https://github.com/Microsoft/msbuild/issues/2999#issuecomment-366078677) like this: `/p:Exclude=\"[coverlet.*]*,[*]Coverlet.Core*\"`, `/p:Include=\"[coverlet.*]*,[*]Coverlet.Core*\"`, or `/p:CoverletOutputFormat=\"json,opencover\"`. + +## Code Coverage + +Enabling code coverage is as simple as setting the `CollectCoverage` property to `true` + +```bash +dotnet test /p:CollectCoverage=true +``` + +After the above command is run, a `coverage.json` file containing the results will be generated in the root directory of the test project. A summary of the results will also be displayed in the terminal. + +## Coverage Output + +Coverlet can generate coverage results in multiple formats, which is specified using the `CoverletOutputFormat` property. For example, the following command emits coverage results in the `opencover` format: + +```bash +dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover +``` + +Supported Formats: + +* json (default) +* lcov +* opencover +* cobertura +* teamcity + +You can specify multiple output formats by separating them with a comma (`,`). + +The output of the coverage result can be specified using the `CoverletOutput` property. + +```bash +dotnet test /p:CollectCoverage=true /p:CoverletOutput='./result.json' +``` + +To specify a directory where all results will be written to (especially if using multiple formats), end the value with a `/`. + +```bash +dotnet test /p:CollectCoverage=true /p:CoverletOutput='./results/' +``` + +### TeamCity Output + +Coverlet can output basic code coverage statistics using [TeamCity service messages](https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages). + +```bash +dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=teamcity +``` + +The currently supported [TeamCity statistics](https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages) are: + +| TeamCity Statistic Key | Description | +| :--- | :--- | +| CodeCoverageL | Line-level code coverage | +| CodeCoverageR | Branch-level code coverage | +| CodeCoverageM | Method-level code coverage | +| CodeCoverageAbsLTotal | The total number of lines | +| CodeCoverageAbsLCovered | The number of covered lines | +| CodeCoverageAbsRTotal | The total number of branches | +| CodeCoverageAbsRCovered | The number of covered branches | +| CodeCoverageAbsMTotal | The total number of methods | +| CodeCoverageAbsMCovered | The number of covered methods | + +## Merging Results + +With Coverlet you can combine the output of multiple coverage runs into a single result. + +```bash +dotnet test /p:CollectCoverage=true /p:MergeWith='/path/to/result.json' +``` + +The value given to `/p:MergeWith` **must** be a path to Coverlet's own json result format. The results in `result.json` will be read, and added to the new results written to by Coverlet. + +## Threshold + +Coverlet allows you to specify a coverage threshold below which it fails the build. This allows you to enforce a minimum coverage percent on all changes to your project. + +```bash +dotnet test /p:CollectCoverage=true /p:Threshold=80 +``` + +The above command will automatically fail the build if the line, branch or method coverage of _any_ of the instrumented modules falls below 80%. You can specify what type of coverage to apply the threshold value to using the `ThresholdType` property. For example to apply the threshold check to only **line** coverage: + +```bash +dotnet test /p:CollectCoverage=true /p:Threshold=80 /p:ThresholdType=line +``` + +You can specify multiple values for `ThresholdType` by separating them with commas. Valid values include `line`, `branch` and `method`. + +By default, Coverlet will validate the threshold value against the coverage result of each module. The `/p:ThresholdStat` option allows you to change this behaviour and can have any of the following values: + +* Minimum (Default): Ensures the coverage result of each module isn't less than the threshold +* Total: Ensures the total combined coverage result of all modules isn't less than the threshold +* Average: Ensures the average coverage result of all modules isn't less than the threshold + +The following command will compare the threshold value with the overall total coverage of all modules: + +```bash +dotnet test /p:CollectCoverage=true /p:Threshold=80 /p:ThresholdType=line /p:ThresholdStat=total +``` + +## Excluding From Coverage + +### Attributes + +You can ignore a method or an entire class from code coverage by creating and applying the `ExcludeFromCodeCoverage` attribute present in the `System.Diagnostics.CodeAnalysis` namespace. + +You can also ignore additional attributes by using the `ExcludeByAttribute` property (short name or full name supported): + +```bash +dotnet test /p:CollectCoverage=true /p:ExcludeByAttribute="Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute" +``` + +### Source Files +You can also ignore specific source files from code coverage using the `ExcludeByFile` property + - Use single or multiple paths (separate by comma) + - Use absolute or relative paths (relative to the project directory) + - Use file path or directory path with globbing (e.g `dir1/*.cs`) + +```bash +dotnet test /p:CollectCoverage=true /p:ExcludeByFile=\"../dir1/class1.cs,../dir2/*.cs,../dir3/**/*.cs\" +``` + +### Filters +Coverlet gives the ability to have fine grained control over what gets excluded using "filter expressions". + +Syntax: `/p:Exclude=[Assembly-Filter]Type-Filter` + +Wildcards +- `*` => matches zero or more characters +- `?` => the prefixed character is optional + +Examples + - `/p:Exclude="[*]*"` => Excludes all types in all assemblies (nothing is instrumented) + - `/p:Exclude="[coverlet.*]Coverlet.Core.Coverage"` => Excludes the Coverage class in the `Coverlet.Core` namespace belonging to any assembly that matches `coverlet.*` (e.g `coverlet.core`) + - `/p:Exclude="[*]Coverlet.Core.Instrumentation.*"` => Excludes all types belonging to `Coverlet.Core.Instrumentation` namespace in any assembly + - `/p:Exclude="[coverlet.*.tests?]*"` => Excludes all types in any assembly starting with `coverlet.` and ending with `.test` or `.tests` (the `?` makes the `s` optional) + - `/p:Exclude=\"[coverlet.*]*,[*]Coverlet.Core*\"` => Excludes assemblies matching `coverlet.*` and excludes all types belonging to the `Coverlet.Core` namespace in any assembly + +```bash +dotnet test /p:CollectCoverage=true /p:Exclude="[coverlet.*]Coverlet.Core.Coverage" +``` + +Coverlet goes a step in the other direction by also letting you explicitly set what can be included using the `Include` property. + +Examples + - `/p:Include="[*]*"` => Includes all types in all assemblies (everything is instrumented) + - `/p:Include="[coverlet.*]Coverlet.Core.Coverage"` => Includes the Coverage class in the `Coverlet.Core` namespace belonging to any assembly that matches `coverlet.*` (e.g `coverlet.core`) + - `/p:Include="[coverlet.*.tests?]*"` => Includes all types in any assembly starting with `coverlet.` and ending with `.test` or `.tests` (the `?` makes the `s` optional) + +Both `Exclude` and `Include` properties can be used together but `Exclude` takes precedence. You can specify multiple filter expressions by separting them with a comma (`,`). + +You can also include coverage of the test assembly itself by setting `/p:IncludeTestAssembly` to `true`. + +### Note for Powershell / VSTS users +To exclude or include multiple assemblies when using Powershell scripts or creating a .yaml file for a VSTS build ```%2c``` should be used as a separator. Msbuild will translate this symbol to ```,```. + +```/p:Exclude="[*]*Examples?%2c[*]*Startup"``` + +VSTS builds do not require double quotes to be unescaped: +``` +dotnet test --configuration $(buildConfiguration) --no-build /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/ /p:Exclude="[MyAppName.DebugHost]*%2c[MyAppNamet.WebHost]*%2c[MyAppName.App]*" +``` + +## SourceLink + +Coverlet supports [SourceLink](https://github.com/dotnet/sourcelink) custom debug information contained in PDBs. When you specify the `--use-source-link` flag in the global tool or `/p:UseSourceLink=true` property in the MSBuild command, Coverlet will generate results that contain the URL to the source files in your source control instead of absolute file paths. diff --git a/Documentation/VSTestIntegration.md b/Documentation/VSTestIntegration.md index 4fc7f635c..bd93a06db 100644 --- a/Documentation/VSTestIntegration.md +++ b/Documentation/VSTestIntegration.md @@ -1,11 +1,13 @@ # Coverlet Integration with VSTest ## Motivation + The cross platform solution for Code Coverage generation for .NET Core projects (in a consistent manner). The asks for Code Coverage support for .NET Core on Linux is the most commented issue on vstest repo: https://github.com/Microsoft/vstest/issues/981#issuecomment-320311552 ## Summary + What would integrating Coverlet with Microsoft Test Platform mean: 1. Coverlet based coverage solution is available as a data collector that does the instrumentation of the necessary modules before test execution and appropriate restore after. @@ -15,6 +17,7 @@ What would integrating Coverlet with Microsoft Test Platform mean: ## Proposed Solution ### Scenarios to support + The following table summarizes the support that needs to be added for a seamless code coverage collection for .NET Core on both Linux and Windows platforms: | Entry point | How will code coverage be enabled? | Syntax | @@ -73,12 +76,16 @@ This runsettings file can easily be provided using command line option as given 2. `dotnet vstest --settings coverletArgs.runsettings` -#### Scope of Enhancement +#### Scope of Enhancement + Currently, advanced options are supported via runsettings. Providing support through additional command line arguments in vstest can be taken up separately. ## Implementation Details -The proposed solution is implemented with the help of [datacollectors](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md). + +The proposed solution is implemented with the help of [datacollectors](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md). + 1. Outproc Datacollector : The outproc collector would always run in a separate process(datacollector.exe/datacollector.dll) than the process in which tests are being executed(testhost*.exe/testhost.dll). This datacollector would be responsible for calling into coverlet APIs for instrumenting dlls, collecting coverage results and sending the coverage output file back to test platform. + 2. Inproc Datacollector : The inproc collector in the testhost process executing the tests. This collector will be needed to remove the dependency on the exit handler to flush the hit files. The datacollectors will be bundled as a separate NuGet package, the reference to which will be added by default in the .NET Core test templates, thus making it the default solution for collecting code coverage for .NET core projects. diff --git a/README.md b/README.md index 8bba58f00..489eeca6a 100644 --- a/README.md +++ b/README.md @@ -1,268 +1,46 @@ -# coverlet [![Build status](https://ci.appveyor.com/api/projects/status/6rdf00wufospr4r8/branch/master?svg=true)](https://ci.appveyor.com/project/tonerdo/coverlet) [![codecov](https://codecov.io/gh/tonerdo/coverlet/branch/master/graph/badge.svg)](https://codecov.io/gh/tonerdo/coverlet) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![NuGet](https://img.shields.io/nuget/v/coverlet.msbuild.svg)](https://www.nuget.org/packages/coverlet.msbuild) +# Coverlet -Coverlet is a cross platform code coverage library for .NET Core, with support for line, branch and method coverage. +Coverlet is a cross platform code coverage framework for .NET, with support for line, branch and method coverage. It works with .NET Framework on Windows and .NET Core on all supported platforms. ## Installation -**Global Tool**: +**VSTest Integration**: ```bash -dotnet tool install --global coverlet.console +dotnet add package coverlet.collector ``` -**Package Reference**: +**MSBuild Integration**: ```bash dotnet add package coverlet.msbuild ``` -## How It Works - -Coverlet generates code coverage information by going through the following process: - -### Before Tests Run - -* Locates the unit test assembly and selects all the referenced assemblies that have PDBs. -* Instruments the selected assemblies by inserting code to record sequence point hits to a temporary file. - -### After Tests Run - -* Restore the original non-instrumented assembly files. -* Read the recorded hits information from the temporary file. -* Generate the coverage result from the hits information and write it to a file. - -_Note: The assembly you'd like to get coverage for must be different from the assembly that contains the tests_ - -## Usage - -Coverlet can be used either as a .NET Core global tool that can be invoked from a terminal or as a NuGet package that integrates with the MSBuild system of your test project. - -### Global Tool - -To see a list of options, run: - -```bash -coverlet --help -``` - -The current options are (output of `coverlet --help`): - -```bash -Cross platform .NET Core code coverage tool 1.0.0.0 - -Usage: coverlet [arguments] [options] - -Arguments: - Path to the test assembly. - -Options: - -h|--help Show help information - -v|--version Show version information - -t|--target Path to the test runner application. - -a|--targetargs Arguments to be passed to the test runner. - -o|--output Output of the generated coverage report - -v|--verbosity Sets the verbosity level of the command. Allowed values are quiet, minimal, normal, detailed. - -f|--format Format of the generated coverage report. - --threshold Exits with error if the coverage % is below value. - --threshold-type Coverage type to apply the threshold to. - --threshold-stat Coverage statistic used to enforce the threshold value. - --exclude Filter expressions to exclude specific modules and types. - --include Filter expressions to include specific modules and types. - --include-directory Include directories containing additional assemblies to be instrumented. - --exclude-by-file Glob patterns specifying source files to exclude. - --exclude-by-attribute Attributes to exclude from code coverage. - --include-test-assembly Specifies whether to report code coverage of the test assembly. - --single-hit Specifies whether to limit code coverage hit reporting to a single hit for each location. - --merge-with Path to existing coverage result to merge. - --use-source-link Specifies whether to use SourceLink URIs in place of file system paths. -``` - -#### Code Coverage - -The `coverlet` tool is invoked by specifying the path to the assembly that contains the unit tests. You also need to specify the test runner and the arguments to pass to the test runner using the `--target` and `--targetargs` options respectively. The invocation of the test runner with the supplied arguments **must not** involve a recompilation of the unit test assembly or no coverage data will be generated. - -The following example shows how to use the familiar `dotnet test` toolchain: - -```bash -coverlet /path/to/test-assembly.dll --target "dotnet" --targetargs "test /path/to/test-project --no-build" -``` - -After the above command is run, a `coverage.json` file containing the results will be generated in the directory the `coverlet` command was run. A summary of the results will also be displayed in the terminal. - -_Note: The `--no-build` flag is specified so that the `/path/to/test-assembly.dll` isn't rebuilt_ - -#### Coverage Output - -Coverlet can generate coverage results in multiple formats, which is specified using the `--format` or `-f` options. For example, the following command emits coverage results in the `opencover` format instead of `json`: - -```bash -coverlet --target --targetargs --format opencover -``` - -Supported Formats: - -* json (default) -* lcov -* opencover -* cobertura -* teamcity - -The `--format` option can be specified multiple times to output multiple formats in a single run: - -```bash -coverlet --target --targetargs --format opencover --format lcov -``` - -By default, Coverlet will output the coverage results file(s) in the current working directory. The `--output` or `-o` options can be used to override this behaviour. - -```bash -coverlet --target --targetargs --output "/custom/path/result.json" -``` - -The above command will write the results to the supplied path, if no file extension is specified it'll use the standard extension of the selected output format. To specify a directory instead, simply append a `/` to the end of the value. - -```bash -coverlet --target --targetargs --output "/custom/directory/" -f json -f lcov -``` - -#### TeamCity Output - -Coverlet can output basic code coverage statistics using [TeamCity service messages](https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages). - -```bash -coverlet --target --targetargs --output teamcity -``` - -The currently supported [TeamCity statistics](https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages) are: - -| TeamCity Statistic Key | Description | -| :--- | :--- | -| CodeCoverageL | Line-level code coverage | -| CodeCoverageR | Branch-level code coverage | -| CodeCoverageM | Method-level code coverage | -| CodeCoverageAbsLTotal | The total number of lines | -| CodeCoverageAbsLCovered | The number of covered lines | -| CodeCoverageAbsRTotal | The total number of branches | -| CodeCoverageAbsRCovered | The number of covered branches | -| CodeCoverageAbsMTotal | The total number of methods | -| CodeCoverageAbsMCovered | The number of covered methods | - -#### Merging Results - -With Coverlet you can combine the output of multiple coverage runs into a single result. - -```bash -coverlet --target --targetargs --merge-with "/path/to/result.json" --format opencover -``` - -The value given to `--merge-with` **must** be a path to Coverlet's own json result format. - -#### Threshold - -Coverlet allows you to specify a coverage threshold below which it returns a non-zero exit code. This allows you to enforce a minimum coverage percent on all changes to your project. - -```bash -coverlet --target --targetargs --threshold 80 -``` - -The above command will automatically fail the build if the line, branch or method coverage of _any_ of the instrumented modules falls below 80%. You can specify what type of coverage to apply the threshold value to using the `--threshold-type` option. For example to apply the threshold check to only **line** coverage: - -```bash -coverlet --target --targetargs --threshold 80 --threshold-type line -``` - -You can specify the `--threshold-type` option multiple times. Valid values include `line`, `branch` and `method`. - -```bash -coverlet --target --targetargs --threshold 80 --threshold-type line --threshold-type method -``` - -By default, Coverlet will validate the threshold value against the coverage result of each module. The `--threshold-stat` option allows you to change this behaviour and can have any of the following values: - -* Minimum (Default): Ensures the coverage result of each module isn't less than the threshold -* Total: Ensures the total combined coverage result of all modules isn't less than the threshold -* Average: Ensures the average coverage result of all modules isn't less than the threshold - -The following command will compare the threshold value with the overall total coverage of all modules: +**Global Tool**: ```bash -coverlet --target --targetargs --threshold 80 --threshold-type line --threshold-stat total +dotnet tool install --global coverlet.console ``` -#### Excluding From Coverage - -##### Attributes +## Quick Start -You can ignore a method or an entire class from code coverage by creating and applying the `ExcludeFromCodeCoverage` attribute present in the `System.Diagnostics.CodeAnalysis` namespace. +### VSTest Integration -You can also ignore additional attributes by using the `ExcludeByAttribute` property (short name or full name supported): +Coverlet is integrated into the Visual Studio Test Platform as a [data collector](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md). To get coverage simply run the following command: ```bash -coverlet --target --targetargs --exclude-by-attribute "Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute" +dotnet test --collect:"XPlat Code Coverage" ``` -##### Source Files +After the above command is run, a `coverage.cobertura.json` file containing the results will be published to the `TestResults` directory as an attachment. A summary of the results will also be displayed in the terminal. -You can also ignore specific source files from code coverage using the `--exclude-by-file` option - - Can be specified multiple times - - Use absolute or relative paths (relative to the project directory) - - Use file path or directory path with globbing (e.g `dir1/*.cs`) +See [documentation](Documentation/VSTestIntegration.md) for advanced usage. -```bash -coverlet --target --targetargs --exclude-by-file "../dir1/class1.cs" -``` - -##### Filters - -Coverlet gives the ability to have fine grained control over what gets excluded using "filter expressions". +_Note: You need to be running .NET Core SDK v2.2.300 and above_ -Syntax: `--exclude '[Assembly-Filter]Type-Filter'` +### MSBuild Integration -Wildcards -- `*` => matches zero or more characters -- `?` => the prefixed character is optional - -Examples - - `--exclude "[*]*"` => Excludes all types in all assemblies (nothing is instrumented) - - `--exclude "[coverlet.*]Coverlet.Core.Coverage"` => Excludes the Coverage class in the `Coverlet.Core` namespace belonging to any assembly that matches `coverlet.*` (e.g `coverlet.core`) - - `--exclude "[*]Coverlet.Core.Instrumentation.*"` => Excludes all types belonging to `Coverlet.Core.Instrumentation` namespace in any assembly - - `--exclude "[coverlet.*.tests?]*"` => Excludes all types in any assembly starting with `coverlet.` and ending with `.test` or `.tests` (the `?` makes the `s` optional) - - `--exclude "[coverlet.*]*" --exclude "[*]Coverlet.Core*"` => Excludes assemblies matching `coverlet.*` and excludes all types belonging to the `Coverlet.Core` namespace in any assembly - -```bash -coverlet --target --targetargs --exclude "[coverlet.*]Coverlet.Core.Coverage" -``` - -Coverlet goes a step in the other direction by also letting you explicitly set what can be included using the `--include` option. - -Examples - - `--include "[*]*"` => Includes all types in all assemblies (everything is instrumented) - - `--include "[coverlet.*]Coverlet.Core.Coverage"` => Includes the Coverage class in the `Coverlet.Core` namespace belonging to any assembly that matches `coverlet.*` (e.g `coverlet.core`) - - `--include "[coverlet.*.tests?]*"` => Includes all types in any assembly starting with `coverlet.` and ending with `.test` or `.tests` (the `?` makes the `s` optional) - -Both `--exclude` and `--include` options can be used together but `--exclude` takes precedence. You can specify the `--exclude` and `--include` options multiple times to allow for multiple filter expressions. - -You can also include coverage of the test assembly itself by specifying the `--include-test-assembly` flag. - -### MSBuild - -In this mode, Coverlet doesn't require any additional setup other than including the NuGet package in the unit test project. It integrates with the `dotnet test` infrastructure built into the .NET Core CLI and when enabled, will automatically generate coverage results after tests are run. - -If a property takes multiple comma-separated values please note that [you will have to add escaped quotes around the string](https://github.com/Microsoft/msbuild/issues/2999#issuecomment-366078677) like this: `/p:Exclude=\"[coverlet.*]*,[*]Coverlet.Core*\"`, `/p:Include=\"[coverlet.*]*,[*]Coverlet.Core*\"`, or `/p:CoverletOutputFormat=\"json,opencover\"`. - -##### Note for Powershell / VSTS users -To exclude or include multiple assemblies when using Powershell scripts or creating a .yaml file for a VSTS build ```%2c``` should be used as a separator. Msbuild will translate this symbol to ```,```. - -```/p:Exclude="[*]*Examples?%2c[*]*Startup"``` - -VSTS builds do not require double quotes to be unescaped: -``` -dotnet test --configuration $(buildConfiguration) --no-build /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/ /p:Exclude="[MyAppName.DebugHost]*%2c[MyAppNamet.WebHost]*%2c[MyAppName.App]*" -``` - -#### Code Coverage - -Enabling code coverage is as simple as setting the `CollectCoverage` property to `true` +Coverlet also integrates with the build system to run code coverage after tests. Enabling code coverage is as simple as setting the `CollectCoverage` property to `true` ```bash dotnet test /p:CollectCoverage=true @@ -270,133 +48,42 @@ dotnet test /p:CollectCoverage=true After the above command is run, a `coverage.json` file containing the results will be generated in the root directory of the test project. A summary of the results will also be displayed in the terminal. -#### Coverage Output - -Coverlet can generate coverage results in multiple formats, which is specified using the `CoverletOutputFormat` property. For example, the following command emits coverage results in the `opencover` format: - -```bash -dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover -``` - -Supported Formats: - -* json (default) -* lcov -* opencover -* cobertura -* teamcity - -You can specify multiple output formats by separating them with a comma (`,`). - -The output of the coverage result can be specified using the `CoverletOutput` property. - -```bash -dotnet test /p:CollectCoverage=true /p:CoverletOutput='./result.json' -``` - -To specify a directory where all results will be written to (especially if using multiple formats), end the value with a `/`. - -```bash -dotnet test /p:CollectCoverage=true /p:CoverletOutput='./results/' -``` - -#### Merging Results +See [documentation](Documentation/MSBuildIntegration.md) for advanced usage. -With Coverlet you can combine the output of multiple coverage runs into a single result. - -```bash -dotnet test /p:CollectCoverage=true /p:MergeWith='/path/to/result.json' -``` - -The value given to `/p:MergeWith` **must** be a path to Coverlet's own json result format. The results in `result.json` will be read, and added to the new results written to by Coverlet. - -#### Threshold - -Coverlet allows you to specify a coverage threshold below which it fails the build. This allows you to enforce a minimum coverage percent on all changes to your project. - -```bash -dotnet test /p:CollectCoverage=true /p:Threshold=80 -``` - -The above command will automatically fail the build if the line, branch or method coverage of _any_ of the instrumented modules falls below 80%. You can specify what type of coverage to apply the threshold value to using the `ThresholdType` property. For example to apply the threshold check to only **line** coverage: - -```bash -dotnet test /p:CollectCoverage=true /p:Threshold=80 /p:ThresholdType=line -``` - -You can specify multiple values for `ThresholdType` by separating them with commas. Valid values include `line`, `branch` and `method`. - -By default, Coverlet will validate the threshold value against the coverage result of each module. The `/p:ThresholdStat` option allows you to change this behaviour and can have any of the following values: - -* Minimum (Default): Ensures the coverage result of each module isn't less than the threshold -* Total: Ensures the total combined coverage result of all modules isn't less than the threshold -* Average: Ensures the average coverage result of all modules isn't less than the threshold - -The following command will compare the threshold value with the overall total coverage of all modules: - -```bash -dotnet test /p:CollectCoverage=true /p:Threshold=80 /p:ThresholdType=line /p:ThresholdStat=total -``` - -#### Excluding From Coverage - -##### Attributes - -You can ignore a method or an entire class from code coverage by creating and applying the `ExcludeFromCodeCoverage` attribute present in the `System.Diagnostics.CodeAnalysis` namespace. - -You can also ignore additional attributes by using the `ExcludeByAttribute` property (short name or full name supported): +### Global Tool -```bash -dotnet test /p:CollectCoverage=true /p:ExcludeByAttribute="Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute" -``` +The `coverlet` tool is invoked by specifying the path to the assembly that contains the unit tests. You also need to specify the test runner and the arguments to pass to the test runner using the `--target` and `--targetargs` options respectively. The invocation of the test runner with the supplied arguments **must not** involve a recompilation of the unit test assembly or no coverage result will be generated. -#### Source Files -You can also ignore specific source files from code coverage using the `ExcludeByFile` property - - Use single or multiple paths (separate by comma) - - Use absolute or relative paths (relative to the project directory) - - Use file path or directory path with globbing (e.g `dir1/*.cs`) +The following example shows how to use the familiar `dotnet test` toolchain: ```bash -dotnet test /p:CollectCoverage=true /p:ExcludeByFile=\"../dir1/class1.cs,../dir2/*.cs,../dir3/**/*.cs\" +coverlet /path/to/test-assembly.dll --target "dotnet" --targetargs "test /path/to/test-project --no-build" ``` -##### Filters -Coverlet gives the ability to have fine grained control over what gets excluded using "filter expressions". - -Syntax: `/p:Exclude=[Assembly-Filter]Type-Filter` +_Note: The `--no-build` flag is specified so that the `/path/to/test-assembly.dll` assembly isn't rebuilt_ -Wildcards -- `*` => matches zero or more characters -- `?` => the prefixed character is optional +See [documentation](Documentation/GlobalTool.md) for advanced usage. -Examples - - `/p:Exclude="[*]*"` => Excludes all types in all assemblies (nothing is instrumented) - - `/p:Exclude="[coverlet.*]Coverlet.Core.Coverage"` => Excludes the Coverage class in the `Coverlet.Core` namespace belonging to any assembly that matches `coverlet.*` (e.g `coverlet.core`) - - `/p:Exclude="[*]Coverlet.Core.Instrumentation.*"` => Excludes all types belonging to `Coverlet.Core.Instrumentation` namespace in any assembly - - `/p:Exclude="[coverlet.*.tests?]*"` => Excludes all types in any assembly starting with `coverlet.` and ending with `.test` or `.tests` (the `?` makes the `s` optional) - - `/p:Exclude=\"[coverlet.*]*,[*]Coverlet.Core*\"` => Excludes assemblies matching `coverlet.*` and excludes all types belonging to the `Coverlet.Core` namespace in any assembly +## How It Works -```bash -dotnet test /p:CollectCoverage=true /p:Exclude="[coverlet.*]Coverlet.Core.Coverage" -``` +Coverlet generates code coverage information by going through the following process: -Coverlet goes a step in the other direction by also letting you explicitly set what can be included using the `Include` property. +### Before Tests Run -Examples - - `/p:Include="[*]*"` => Includes all types in all assemblies (everything is instrumented) - - `/p:Include="[coverlet.*]Coverlet.Core.Coverage"` => Includes the Coverage class in the `Coverlet.Core` namespace belonging to any assembly that matches `coverlet.*` (e.g `coverlet.core`) - - `/p:Include="[coverlet.*.tests?]*"` => Includes all types in any assembly starting with `coverlet.` and ending with `.test` or `.tests` (the `?` makes the `s` optional) +* Locates the unit test assembly and selects all the referenced assemblies that have PDBs. +* Instruments the selected assemblies by inserting code to record sequence point hits to a temporary file. -Both `Exclude` and `Include` properties can be used together but `Exclude` takes precedence. You can specify multiple filter expressions by separting them with a comma (`,`). +### After Tests Run -You can also include coverage of the test assembly itself by setting `/p:IncludeTestAssembly` to `true`. +* Restore the original non-instrumented assembly files. +* Read the recorded hits information from the temporary file. +* Generate the coverage result from the hits information and write it to a file. -### SourceLink +_Note: The assembly you'd like to get coverage for must be different from the assembly that contains the tests_ -Coverlet supports [SourceLink](https://github.com/dotnet/sourcelink) custom debug information contained in PDBs. When you specify the `--use-source-link` flag in the global tool or `/p:UseSourceLink=true` property in the MSBuild command, Coverlet will generate results that contain the URL to the source files in your source control instead of absolute file paths. +## Cake Add-In -### Cake Addin -If you're using [Cake Build](https://cakebuild.net) for your build script you can use the [Cake.Coverlet](https://github.com/Romanx/Cake.Coverlet) addin to provide you extensions to dotnet test for passing coverlet arguments in a strongly typed manner. +If you're using [Cake Build](https://cakebuild.net) for your build script you can use the [Cake.Coverlet](https://github.com/Romanx/Cake.Coverlet) add-in to provide you extensions to dotnet test for passing Coverlet arguments in a strongly typed manner. ## Issues & Contributions