Unable to get coverlet to work with vstest

[nzain]

I'm using coverlet.msbuild on several modern netstandard2.0 nunit test projects. A simple dotnet test Some.csproj /p:CollectCoverage=true together with smart Directory.Build.Targets like this

<?xml version="1.0" encoding="utf-8"?>
<Project>
  <PropertyGroup>
    <CoverletOutputFormat>opencover</CoverletOutputFormat>
    <CoverletOutput>$(MSBuildThisFileDirectory)TestResults/Coverage_$(AssemblyName).xml</CoverletOutput>
  </PropertyGroup>
</Project>

creates SonarQube compatible coverage files. This works really well.

However, one of my test projects is ugly, has many dependencies (full net472 framework), and a ton of native libraries as a bonus. The tests are important, but I don't own the dependencies and the owners are very conservative - so I can't change this situation right now. In this case dotnet test doesn't work, but dotnet vstest does work fine (and the vstest.console.exe as well).

Problem: I can't get coverlet to work with VSTest. Following coverlet's Readme.md I referenced coverlet.collector instead of its msbuild brother. Since I have to target a dll instead of a project, I have to build first. I'm using msbuild here, not dotnet publish for the same reasons dotnet test doesn't work. Then dotnet vstest SomeUglyTest.dll runs my tests as expected. However, all of the following attempts to collect coverage failed:

1. dotnet vstest SomeUglyTest.dll --settings:coverletArgs.runsettings (with a unmodified example from somewhere here in the repository) prints the warning Data collection : Could not find data collector 'XPlat Code Coverage' and obviously doesn't create any coverage output.
2. dotnet vstest SomeUglyTest.dll --collect:"XPlat Code Coverage" prints the same warning, no output file.

related: #395 #190

vstest is very difficult to understand... but I think that it requires some kind of "collector.dll", which should be satisfied by referencing the coverlet.collector nuget package. However, just referencing the package does not mean msbuild will copy something to bin\debug\. In fact, I haven't found such a dll. dotnet publish might do things differently, but I can't use it.

Any ideas?

[MarcoRossignoli]

If you cannot touch project dependency I propose to use .net tool and not collectors/msbuild.
https://github.com/tonerdo/coverlet/blob/master/Documentation/GlobalTool.md#code-coverage

something like

F:\git\coverletissues\issue595\FullFrameworkTest
λ msbuild /v:m
Microsoft (R) Build Engine version 16.3.0-preview-19456-02+ee8294b55 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

  ClassLibrary1 -> F:\git\coverletissues\issue595\FullFrameworkTest\ClassLibrary1\bin\Debug\ClassLibrary1.dll
  TestLib -> F:\git\coverletissues\issue595\FullFrameworkTest\TestLib\bin\Debug\TestLib.dll

F:\git\coverletissues\issue595\FullFrameworkTest
λ coverlet TestLib\bin\Debug\TestLib.dll --target "dotnet" --targetargs "vstest TestLib\bin\Debug\TestLib.dll"
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1,0110 Seconds

Calculating coverage result...
  Generating report 'F:\git\coverletissues\issue595\FullFrameworkTest\coverage.json'
+---------------+------+--------+--------+
| Module        | Line | Branch | Method |
+---------------+------+--------+--------+
| ClassLibrary1 | 100% | 100%   | 100%   |
+---------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | 100% | 100%   | 100%   |
+---------+------+--------+--------+

[MarcoRossignoli]

BTW if you hit know issue https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test using .net tool driver, you must use collectors.
If you don't have issue with msbuild that suffer of same problem I think it's better use .net tool.

The approach you suggested is doable but ugly/fragile at the moment.
We need to provide collector libs and runsettings.
So you need to manual(or script) copy https://www.nuget.org/packages/coverlet.collector/ libs and provide a runsettings, I did it copying dlls on output build folder near to tested lib.

F:\git\coverletissues\issue595\FullFrameworkTest\TestLib
λ dotnet vstest bin\Debug\TestLib.dll --Settings:"F:\git\coverletissues\issue595\FullFrameworkTest\TestLib\runsettings.xml"
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Attachments:
  F:\git\coverletissues\issue595\FullFrameworkTest\TestLib\TestResults\c28b0e05-f46a-4d17-be2f-e8ee7a255887\coverage.cobertura.xml
Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1,1004 Seconds

Runsettings(there is an open bug on vstest at the moment..so we need to specify inproc collector by hand microsoft/vstest#2205)

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
  <InProcDataCollectionRunSettings>
    <InProcDataCollectors>
      <InProcDataCollector assemblyQualifiedName="Coverlet.Collector.DataCollection.CoverletInProcDataCollector, coverlet.collector, Version=1.1.0.0, Culture=neutral, PublicKeyToken=null"
                     friendlyName="XPlat Code Coverage"
                     enabled="True"
                     codebase="coverlet.collector.dll" />
    </InProcDataCollectors>
  </InProcDataCollectionRunSettings>
</RunSettings>

[nzain]

That night I thought that I might try to migrate all related projects to the new csproj format. This made dotnet test and coverlet coverage work, I don't have to fiddle with vstest, plus projects are cleaner and dependencies are transitive. Thanks for your quick response and suggestion, nevertheless!