run coverlet as global tool with vstest

[Nusserdt]

I try to run coverlet as global tool with vstest. My command line is:

coverlet {path to test .dll} --target "dotnet" --targetargs "vstest {path to test .dll} --logger:trx" --format cobertura --threshold 80 --verbosity detailed

coverlet response:

..
Hits file:'C:\Users\user\AppData\Local\Temp\Pom2_98c0d86c-7f21-4f7a-9ba5-f3690201c1ae' not found for module: 'Pom2'
Hits file:'C:\Users\user\AppData\Local\Temp\Services_98c0d86c-7f21-4f7a-9ba5-f3690201c1ae' not found for module: 'Services'
Hits file:'C:\Users\user\AppData\Local\Temp\Visualization_98c0d86c-7f21-4f7a-9ba5-f3690201c1ae' not found for module: 'Visualization'
..

for every single referenced assemblies that coverlet collect before:

..
Instrumented module: 'D:\IMSHH\IMSHH\bin\ptc\Release\Pom2.dll'
Instrumented module: 'D:\IMSHH\IMSHH\bin\ptc\Release\Services.dll'
Instrumented module: 'D:\IMSHH\IMSHH\bin\ptc\Release\Visualization.dll'
..

coverlet version: Cross platform .NET Core code coverage tool
1.5.0.0 (installed is dotnet tool install --global coverlet.console --version 1.5.3)

Whats going wrong here?

[MarcoRossignoli]

We've issue with Unload event https://github.com/tonerdo/coverlet/blob/5a139b200f7bf51c21660b93f41906f4571c971b/src/coverlet.core/Coverage.cs#L231 that will be resolved with collectos.

But before some question(maybe silly)

1. Do you run tests that include that libs?
2. Same result with test tools(no vstest) https://github.com/tonerdo/coverlet#global-tool?
3. If you try with msbuild or collector way result are the same?

In case can you provide a repro(or project is open sorce and we can clone)?

[Nusserdt]

Hello @MarcoRossignoli thanks for your response.

1. No, most of them (>100) looks like indirect depedencies, if there is an option to analyse only direct depedencies, I would prefer that.

2. Yes I get the same results with
   coverlet {path to test .dll} --target "dotnet" --targetargs "test {path to test .dll} --logger:trx" --format cobertura --threshold 80 --verbosity detailed
   and
   coverlet {path to test .dll} --target "dotnet" --targetargs "--logger:trx" --format cobertura --threshold 80 --verbosity detailed too

3. Unfortunately I can't share the repository or an project. I know it would be much easier to found out whats going on, sorry.

[MarcoRossignoli]

No, most of them (>100) looks like indirect depedencies, if there is an option to analyse only direct depedencies, I would prefer that.

The issue could be related to this...if you don't use that libs you should exclude from coverage(or better include only dll you're testing)...we don't have at the moment a way to exclude "unused" dll.
We've some on going work on it(actually stale) #225

[Nusserdt]

That means I have to create a list with all unused .dll and pass them in a array to the --exclude-by-file variable? For example:

--exclude-by-file @("Pom2.dll", "Services.dll")

[MarcoRossignoli]

yep, global tool filter sample are here https://github.com/tonerdo/coverlet/blob/master/Documentation/GlobalTool.md#filters
Or better maybe use "include" and specify only libs you want.

[Nusserdt]

@MarcoRossignoli I try to switch to coverlet Version 1.2.0.0 and remarkable is that the coverlet execution stops immediately with the message:

Symbols were found but are not matching the assembly

What should that tell me?

[MarcoRossignoli]

Mmm can you take a screenshot of command line?I don't understand if it's related to coverlet (seem that pdb are not aligned with compilation, try also to run git clean -fdx from repo root before).
I mean that isn't a coverlet message.

[Nusserdt]

@MarcoRossignoli Sorry that I have to mask some parts.

I try both:
coverlet UnitTests.dll --target "vstest.console" --targetargs "project\Release\UnitTests.dll --settings:C:\Users\user\Desktop\CoverageTests\TestSettings.runsettings --logger:trx" --format cobertura --threshold 80

and

coverlet UnitTests.dll --target "dotnet" --targetargs "vstest project\Release\UnitTests.dll --settings:C:\Users\user\Desktop\CoverageTests\TestSettings.runsettings --logger:trx" --format cobertura --threshold 80

it also fails without the --settings property

[MarcoRossignoli]

Can you add coverlet logs? https://github.com/tonerdo/coverlet/blob/master/Documentation/Troubleshooting.md#coverlet-global-tool

[MarcoRossignoli]

@MarcoRossignoli I try to switch to coverlet Version 1.2.0.0 and remarkable is that the coverlet execution stops immediately with the message:

Have installed dotnet tool install --global coverlet.console --version 1.5.3?

[Nusserdt]

@MarcoRossignoli As you knew before coverlet 1.2.0.0 returns on

coverlet UnitTests.dll --target "dotnet" --targetargs "test UnitTests.dll --no-build" --verbosity detailed

--> Unrecognized option '--verbosity'

So I do that with coverlet.console --version 1.5.3 and the detailed verbosity log is splitted in 3 parts:

1. Part Excluded module filter '[xunit*]*'

• for each dll of these over 100 dlls (I think there are all referenced by the test-dll) stand the line Instrumented module: 'path of dll'

• between that, you found about 3 warning lines with: Unable to instrument module: TableWPF.dll because : Failed to resolve assembly: 'Framework, Version=4.0.0.0, Culture=neutral, PublicKeyToken=13123214134'

• in the end of part 1 I got an UnitTests.dll(1,1): error MSB4025: The project file could not be loaded. Data at the root level is invalid. Line 1, position 1.

2. Part Calculating coverage result...

• equivalent to the amount of lines before, I get for each the message: Hits file:'ToolStrip_8bbb009a-0a79-4bc5-8445-127004140251' not found for module: 'ToolStrip'

• here, all lines are the same

3. Part Generating report 'coverage.json'

+--------------------------------------------------------------+------+--------+--------+
| Module | Line | Branch | Method |
+--------------------------------------------------------------+------+--------+--------+
| Module A | 0% | 0% | 0% |
+--------------------------------------------------------------+------+--------+--------+
| Module B | 0% | 0% | 0% |
+--------------------------------------------------------------+------+--------+--------+
| Module C | 0% | 0% | 0% |
+--------------------------------------------------------------+------+--------+--------+
| Module D | 0% | 100% | 0% |

• most entries of the table of Line, Branch and Method coverage have 0%, but in some rare modules there stand 100% Branch coverage

• at the end of this part there is a summery with the average coverage (look like he ignoring the 100% Branch coverage before)

+---------+------+--------+--------+
| | Line | Branch | Method |
+---------+------+--------+--------+
| Total | 0% | 0% | 0% |
+---------+------+--------+--------+
| Average | 0% | 0% | 0% |
+---------+------+--------+--------+

To summarize:

I think Part 2 and 3 of the log have nothing to do with my issue. But maybe the MSB4025 is related?

If I open these particular UnitTest project (*.vbproj) everything is fine, the only error I get is:
Solution is not saved. Please save your solution before managing NuGet packages.
If I save the *.sln I can rebuild without any errors.

[MarcoRossignoli]

Ok so I infer that this is a .NET framework project(no core) written in vb.
Can you try to use filter(--include) https://github.com/tonerdo/coverlet/blob/master/Documentation/GlobalTool.md#filters to include only your dll(and exclude references that could lead to problems) and make sure that all dll are present in bin\Debug|Release folder(remember git -fdx before to cleanup repo, to avoid double instrumenting if previous commands failed)?

[Nusserdt]

@MarcoRossignoli One question for my general understanding. Which dll should be include for coverlet? All references I found inside the Project References?

Is it important also include System References?

Or should I only include the references which are the results from the imports of the test classes?

Thanks a lot for your help.

[MarcoRossignoli]

You should include only "your code" for instance suppose to have 3 project A,B,C that generates assemblies A.dll B.dll C.dll and suppose that B and C are libs and A is a WinForm project references B/C and System.Windows.Form(and other system libs). You've also a test project called D.dll.
In this case you should add coverlet package to D and run with filter --include "[A]*,[B]*,[C]*"(all types defined in assemblies) in this way you're instrumenting only "your code" and not references that coverlet will found in bin\Debug|Release folder, you should confirm that using --verbosity detailed switch and read in console that only ABC libs are Instrumented module: 'path of dll'
for ref https://github.com/tonerdo/coverlet/blob/master/Documentation/GlobalTool.md#filters

[Nusserdt]

@MarcoRossignoli thanks a lot for you explanation, that helps a lot. I think I found something importent about this issue. First of all, I understand now that my particular UnitTest project only works with vstest/vstest.console. test or mstest fails and don't work. (I have a kind of an idea why, but it's hard to explain)

---

so running
vstest.console Project.UnitTests.dll /Logger:trx
works exactly like expected. All Test get found, executed an passed.

running
dotnet vstest Project.UnitTests.dll /Logger:trx
returns Project.UnitTests.dll no test is available. Make sure that Testdiscoverer and -executor are registered and the version settings for platform and framework are correct, and then try again.

and that is exactly the feedback what I get, when I am running:
coverlet Project.UnitTests.dll --target "dotnet" --targetargs "vstest Project.UnitTests.dll --logger:trx" --verbosity detailed --include "[Project]*"

---

so obviously dotnet vstest can't find any tests inside the Project.UnitTests.dll but why vstest.console can? Is there a way to run coverlet with vstest.console? Do you have an idea how I can investigate this issue?

[MarcoRossignoli]

I think you can try with coverlet Project.UnitTests.dll --target "vstest.console" --targetargs "Project.UnitTests.dll /Logger:trx" --verbosity detailed --include "[Project]*"
Dotnet tool run --target as process and pass --targetargs https://github.com/tonerdo/coverlet/blob/master/src/coverlet.console/Program.cs#L80 and before it instruments module(Project.UnitTests.dll) and when process closes(tests run ends) collect the coverage.
In this way we should have:

1. coverlet Project.UnitTests.dll --verbosity detailed --include "[Project]*" -> instrumentation of dlls
2. Run command vstest.console --targetargs Project.UnitTests.dll /Logger:trx on external process
3. Collect coverage after external process end.

[Nusserdt]

@MarcoRossignoli with:
coverlet Project.UnitTests.dll --target "vstest.console" --targetargs "Project.UnitTests.dll /Logger:trx" --verbosity detailed --include "[Project]*"

I get the error message (in red):
The system cannot find the specified file.

after Step 1 I think. Coverlet intrument/exclude the dlls before

Seems like dotnet don't know vstest.console

[MarcoRossignoli]

Did some test on my repro...it works(missed .exe)

C:\git\marcorossignoli.github.io\src\coverlet\Issue0 (int -> origin)
λ coverlet "C:\git\marcorossignoli.github.io\src\coverlet\Issue0\CoverletTest\bin\Debug\netcoreapp2.1\CoverletTest.dll" --target "vstest.console.exe" --targetargs "C:\git\marcorossignoli.github.io\src\coverlet\Issue0\CoverletTest\bin\Debug\netcoreapp2.1\CoverletTest.dll /Logger:trx" --verbosity detailed
Excluded module filter '[xunit*]*'
Instrumented module: 'C:\git\marcorossignoli.github.io\src\coverlet\Issue0\CoverletTest\bin\Debug\netcoreapp2.1\CoverletSampleLib.dll'
Excluded module: 'C:\git\marcorossignoli.github.io\src\coverlet\Issue0\CoverletTest\bin\Debug\netcoreapp2.1\xunit.runner.reporters.netcoreapp10.dll'
Excluded module: 'C:\git\marcorossignoli.github.io\src\coverlet\Issue0\CoverletTest\bin\Debug\netcoreapp2.1\xunit.runner.utility.netcoreapp10.dll'
Excluded module: 'C:\git\marcorossignoli.github.io\src\coverlet\Issue0\CoverletTest\bin\Debug\netcoreapp2.1\xunit.runner.visualstudio.dotnetcore.testadapter.dll'
Microsoft (R) Test Execution Command Line Tool Version 16.3.0-preview-20190715-02
Copyright (c) Microsoft Corporation.  All rights reserved.
Starting test execution, please wait...
[xUnit.net 00:00:00.5781395]   Discovering: CoverletTest
[xUnit.net 00:00:00.6422694]   Discovered:  CoverletTest
[xUnit.net 00:00:00.6484240]   Starting:    CoverletTest
[xUnit.net 00:00:00.8154479]   Finished:    CoverletTest
  V CoverletTest.MyFunTest.Test2 [9ms]
  V CoverletTest.TestClass.Test [9ms]
Results File: C:\git\marcorossignoli.github.io\src\coverlet\Issue0\TestResults\Marco_2019-08-14_11_49_48.trx
Test Run Successful.
Total tests: 2
     Passed: 2
 Total time: 1,9266 Seconds

Calculating coverage result...
  Generating report 'C:\git\marcorossignoli.github.io\src\coverlet\Issue0\coverage.json'
+-------------------+------+--------+--------+
| Module            | Line | Branch | Method |
+-------------------+------+--------+--------+
| CoverletSampleLib | 100% | 100%   | 100%   |
+-------------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | 100% | 100%   | 100%   |
+---------+------+--------+--------+

Remeber to build the test/solution before

[Nusserdt]

@MarcoRossignoli tears in my eyes. That was the missing part of the puzzle.

Finally
coverlet Project.UnitTests.dll --target "vstest.console.exe" --targetargs "Project.UnitTests.dll /Logger:trx" --verbosity detailed --include "[Project]*"
works well and returns solid results.

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 89,92% | 82,12% | 90,82% |
+---------+--------+--------+--------+
| Average | 89,92% | 82,12% | 90,82% |
+---------+--------+--------+--------+

Thanks for your patience, keep going for your great project.

[MarcoRossignoli]

Glad to hear!Feel free to close the issue!

keep going for your great project.

Credits goes to all contributors and to @tonerdo's idea.