Integrate coverlet with VSTest (Microsoft Test Platform)

[PBoraMSFT]

What would integrating Coverlet with Microsoft Test Platform mean:

1. Coverlet based coverage solution is available as a data collector that does the instrumentation of the necessary modules before test execution and appropriate restore after.
2. Authoring – When new test projects are created (dotnet mstest), references to the data collector package are added by default. This reduces adoption friction for customers.
3. Test execution - Today Coverlet is invoked as a msbuild target and as such only works with dotnet test (and requires a csproj). With the data collector in bullet 1 becoming available, coverage can also be collected when tests are run on built binaries (dotnet vstest)

Edit: Fixed markdown typos. I was referring to bullet 1 in the above description, not the issue number.

[MarcoRossignoli]

With the data collector in 1 becoming available,

Can we see the implementation?
Could this integration help with this issue #329 ?We're trying to find a way to integrate with VSTest but with a good user experience(RunSettings file is not the best maybe)

Today Coverlet is invoked as a msbuild target and as such only works with dotnet test (and requires a csproj).

We have also dotnet tool, btw integration with dotnet vstest would be great.

We're working on improving/reorganize "internals", this should lead to "stable"(cross finger) interface/contracts usable "outside" as standalone api.
Maybe to integrate with VSTest we need to "refactor"(separate reports and engine) even more and ship coverlet.core.dll as nuget package, that is in line with current refactor plan.

cc: @tonerdo

[vagisha-nidhi]

@MarcoRossignoli Adding implementation details.

Scenario Explanation :

1. Integration of coverlet such that it works with dotnet test.
   a. Command line invocation will be : dotnet test --collect:"Xplat Code Coverage"
2. There will be an outproc datacollector[CoverletDataCollector] with friendly name "Xplat Code Coverage" which will have a dependency on coverlet.core to get coverage data by means of coverlet.core APIs such as coverage.PrepareModules(), coverage.GetCoverageResult().
3. CoverletDataCollector will be responsible for creating result attachments from CoverageResult object and send coverage result back to test platform using DataCollectionSink.
4. CoverletDataCollector will be a separate Nuget package which can be consumed by .Net Core unit test projects.
5. CoverletDataCollector will have a dependency on coverlet.core.
6. Unit test projects templates will have a package reference to CoverletDataCollector.

<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="1.3.2" />
    <PackageReference Include="MSTest.TestFramework" Version="1.3.2" />
    <PackageReference Include="CoverletDataCollector" Version="1.0.0" />   
  </ItemGroup>

Asks :

1. CoverletDataCollector as a separate nuget package. We could contribute this to coverlet repo.
   This will have a dependency on coverlet.core.
2. The issue Spurious 'Unable to read beyond end of stream' in CI while collecting results #210 will be addressed alongside as --collect:"Xplat Code Coverage" will add inproc datacollector to runsettings and get this can be picked up from
   CoverletDataCollector nuget.
3. Getting [WIP] Implement a VSTest data collector to write coverage data to disk #329 PR to completion and getting the
   inprocdatacollector as part of CoverletDataCollector Nuget package.

[MarcoRossignoli]

Some questions(so you can save me/us some time on docs):

CoverletDataCollector will be responsible for creating result attachments from CoverageResult object and send coverage result back to test platform using DataCollectionSink.

What "results" should send?Coverage output file i.e. cobertura file etc...?

CoverletDataCollector will be a separate Nuget package which can be consumed by .Net Core unit test projects.

How versioning works?I mean if we publish collector package and we need to update(collector self or some transient dependencies) due to bug fix new features, is unit test projects template versionable?

Getting #329 PR to completion and getting the
inprocdatacollector as part of CoverletDataCollector Nuget package.

It's not clear to me why/when use InProcess collector vs out of process one(public class NewDataCollector : DataCollector if I understood correctly), and how out of process can work.
Coverlet need to flush hits files before calculate coverage...at the moment we rely on exit events that seem to lead to issue(corrupted file) when for delay reason VSTest runtime decide to kill process. If we need to call that flush(UnloadModule) method we need to be in-process, do we need to implement some interprocess comunication for out of process collector?

[tonerdo]

Thanks for elaborating @vagisha-nidhi. I'm happy you're looking to make Coverlet part of the out of the box experience, we definitely intend to work through this with you. Your implementation details seem fine to me. However, I have a few of questions:

Command line invocation will be : dotnet test --collect:"Xplat Code Coverage"

How will users pass extra arguments like coverage format, threshold etc.

CoverletDataCollector as a separate nuget package.

Can this be called coverlet.collector instead? Just to keep it consistent with existing package names

For my learning benefit

What's the difference between an in-proc and out-of-proc datacollector? How does one create both? Which one do we need to make this integration work?

[tonerdo]

@MarcoRossignoli

What "results" should send?Coverage output file i.e. cobertura file etc...?

I believe it'll be the coverage output file. See here: https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md#datacollectionsink

[cltshivash]

Some questions(so you can save me/us some time on docs):

CoverletDataCollector will be responsible for creating result attachments from CoverageResult object and send coverage result back to test platform using DataCollectionSink.

What "results" should send?Coverage output file i.e. cobertura file etc...?

It will be the coverage output file.

CoverletDataCollector will be a separate Nuget package which can be consumed by .Net Core unit test projects.

How versioning works?I mean if we publish collector package and we need to update(collector self or some transient dependencies) due to bug fix new features, is unit test projects template versionable?

There's no versioning for the test template. If a new collector package is pushed, the template will be updated with the newer version. The existing users can update the nuget package to get the latest changes.

Getting #329 PR to completion and getting the
inprocdatacollector as part of CoverletDataCollector Nuget package.

It's not clear to me why/when use InProcess collector vs out of process one(public class NewDataCollector : DataCollector if I understood correctly), and how out of process can work.
Coverlet need to flush hits files before calculate coverage...at the moment we rely on exit events that seem to lead to issue(corrupted file) when for delay reason VSTest runtime decide to kill process. If we need to call that flush(UnloadModule) method we need to be in-process, do we need to implement some interprocess comunication for out of process collector?

We will need both the collectors. The inproc collector (which runs in the testhost process) is needed for the flushing of the hit files to be reliable and not dependent on the exit handler. The outproc collector is needed to setup the test session (instrument the dlls)

[cltshivash]

Thanks @tonerdo and @MarcoRossignoli for reviewing the enhancement and the comments. We are excited to work with you to enable this integration as it would address the code coverage ask in a seamless way with dotnet test/vstest.

Thanks for elaborating @vagisha-nidhi. I'm happy you're looking to make Coverlet part of the out of the box experience, we definitely intend to work through this with you. Your implementation details seem fine to me. However, I have a few of questions:

Command line invocation will be : dotnet test --collect:"Xplat Code Coverage"

How will users pass extra arguments like coverage format, threshold etc.

There are options to pass runsettings information through cmdline. https://github.com/Microsoft/vstest-docs/blob/master/docs/configure.md. This doesn't work well for datacollector settings under which the coverage format, threshold would fall under. As of now these options would need a runsettings file to specify a value other than the default. We will look addressing the gap here.

CoverletDataCollector as a separate nuget package.

Can this be called coverlet.collector instead? Just to keep it consistent with existing package names

Yes.

For my learning benefit

What's the difference between an in-proc and out-of-proc datacollector? How does one create both? Which one do we need to make this integration work?

The inproc data collector runs in the testhost process executing the tests (testhost*.exe/ testhost.dll). The outproc collector would run in a separate process (datacollector.exe/datacollector.dll). We would need the inproc collector to remove the dependency on the exit handler to flush the hits and would need the outproc collector to setup the test run session (instrument dlls) and restore.

[MarcoRossignoli]

To start to work on it could be good if some of you(MS guys) write a documentation guide under Documentation/VSTestIntegration.md where we(coverlet guys) can see how is the user experience with dotnet test/vstest and setting files for all types of parameters coverlet uses.
This would be useful for two reason, first we'll have some user documentation ready and we can understand how good/bad is, second allow us to understand if we need to rethink someting, for instance if the tracker generation is ok ASIS.
What do you think @tonerdo cc @petli to keep him up to date.

[vagisha-nidhi]

@MarcoRossignoli That would be a great start. I'll shortly raise a PR on the repo with the documentation. We can discuss the next steps further.

[ViktorHofer]

@vagisha-nidhi What is missing here and when will we be able to try this out with dotnet test?

[MarcoRossignoli]

FYI @ViktorHofer #410 (comment)