Using VSTest How Is It Possible To Display Code Coverage In Console Output So That Gitlab CI And Equivalent CI Servers Can Parse The Output To Produce A Coverage Badge?

[dcs3spp]

Hi,

Is it possible to get coverage displayed in console output logs when using VSTest? I am using GitLab CI server. This allows developers to parse console output from a pipeline test job to produce a coverage badge.

My current VSTest command and coverlet settings are displayed below:

VSTest

dotnet test --settings:${workspaceFolder}/Tests/coverlet.runsettings --collect:"XPlat Code Coverage" WebApp.sln

Coverlet run settings

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura</Format>          
          <Exclude>
            [Bunit.*]*,
            [WebApp.Repository]Cyotek.Collections.Generic.CircularBuffer,
            [Humanizer]*,
            [Minio.*]Minio.*,
            [System.Reactive]*,
            [WebApp.Testing.Utils]*,
            [WebApp]WebApp.Program,
            [WebApp.BackgroundServices]WebApp.BackgroundServices.Program,
            [WebApp.Views]WebApp.Pages.Pages__Host
            [xunit.*],
          </Exclude>
          <IncludeDirectory>../Src/</IncludeDirectory>
          <IncludeTestAssembly>false</IncludeTestAssembly>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

This outputs coverage cobertura files in TestResults file per project, which I am merging using ReportGenerator. I am using a run settings file to facilitate specifying IncludeDirectory and Exclude assemblies / third party classes. These coverlet features help reduce command length and complexity for use in CI jobs. Furthermore, it allows me to specify the solution file rather than run a test command for each individual project.

However, it does not look as though coverage is output on the console, which is required for generating coverage badges for use in CI pipeline services, such as GitLab.

The only way that I can get coverage displayed in console logs is to use coverlet.msbuild. However, I am finding it difficult to replicate the equivalent settings from VSTest. Thus, I am getting a different set of coverage results.

msbuild command

dotnet test \
/p:CollectCoverage=true \
/p:CoverletOutput='./TestResults/' \
/p:CoverletOutputFormat=cobertura \
/p:Exclude=\"[Bunit.*]*,[WebApp.Repository]Cyotek.Collections.Generic.CircularBuffer,[Humanizer]*,[Minio.*]Minio.*,[System.Reactive]*,[WebApp.Testing.Utils]*,[WebApp]WebApp.Program,[WebApp.BackgroundServices]WebApp.BackgroundServices.Program,[WebApp.Views]WebApp.Pages.Pages__Host,[xunit.*]\" \
/p:IncludeTestAssembly=false \
WebApp.sln

The above command replicates most configuration items apart from IncludeDirectory.

1. Is it possible to output code coverage statistics to console using coverlet with VSTest
2. Is it possible to have coverlet.msbuild read from a run settings file like it's equivalent VSTest command?
3. Is there an equivalent option for VSTest IncludeDirectory when using coverlet.msbuild?

Alternatively, I am investigating the possibility of sticking with current VSTest approach and checking if reportgenerator allows generating output to a similar format to msbuild.coverlet. I could then output this to the console allowing the CI server to parse the coverage for use in the badge.

[petli]

The VSTest approach (which is recommended especially in CI situations due to limitations with the msbuild or CLI drivers) cannot output a summary, though perhaps the extensions to VSTest mentioned in the discussion around #1015 might make that possible in future versions.

In the meantime, if you already process the files with ReportGenerator you can maybe use one of the several summary report types or the badge report type to get the output you need, as well as the merged coverage file:

From https://github.com/danielpalme/ReportGenerator:

   Report types:       The output formats and scope (separated by semicolon).
                       Values: Badges, Clover, Cobertura, CsvSummary, Html, HtmlChart, HtmlInline, 
                       HtmlInline_AzurePipelines, HtmlInline_AzurePipelines_Dark, HtmlSummary,
                       JsonSummary, Latex, LatexSummary, lcov, MHtml, PngChart, SonarQube, TeamCitySummary,
                       TextSummary, Xml, XmlSummary

[dcs3spp]

Thanks @petli,

In process of investigating reportgenerator with Badges and TextSummary.
Like the idea of #1015....

Many thanks, appreciated :)