Solution-wide coverage results are saved under random project

[aarndt]

This is in regard to the automatic merging of coverage files when tests are run at the solution-level, as described in https://devblogs.microsoft.com/dotnet/whats-new-in-our-code-coverage-tooling/

Auto-Merge for solutions

Running dotnet test --collect "Code Coverage" at the solution level now automatically merges code coverage for all your test projects. Visit Scenario 24 Code coverage for solution to see full example.

I pulled the referenced repo with sample code, and while the project-level coverage files were merged into a single file, the project/folder where the file was saved differed between test runs.

dotnet test --collect "Code Coverage"    

Microsoft (R) Test Execution Command Line Tool Version 17.5.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     4, Skipped:     0, Total:     4, Duration: 7 ms - Calculator.Core.Tests.dll (net7.0)
Passed!  - Failed:     0, Passed:     4, Skipped:     0, Total:     4, Duration: 512 ms - Calculator.Console.Tests.dll (net7.0)
Passed!  - Failed:     0, Passed:     4, Skipped:     0, Total:     4, Duration: 8 ms - Calculator.Server.Tests.dll (net7.0)
  Skipped TestOperations("/subtract/3/5","-2") [4 s]
  Skipped TestOperations("/add/3/5","8") [4 s]
  Skipped TestOperations("/multiply/3/5","15") [4 s]
  Skipped TestOperations("/divide/35/5","7") [4 s]

Skipped! - Failed:     0, Passed:     0, Skipped:     4, Total:     4, Duration: 4 s - Calculator.Server.IntegrationTests.dll (net7.0)

Attachments:
  C:\Users\MyUser\codecoverage\samples\Calculator\tests\Calculator.Server.IntegrationTests\TestResults\ab55596d-baa4-4bb2-9038-16634355968f\MyUser_2024-01-01.16_44_54.coverage

2nd execution

dotnet test --collect "Code Coverage"    

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     4, Skipped:     0, Total:     4, Duration: 7 ms - Calculator.Core.Tests.dll (net7.0)
Passed!  - Failed:     0, Passed:     4, Skipped:     0, Total:     4, Duration: 505 ms - Calculator.Console.Tests.dll (net7.0)
Passed!  - Failed:     0, Passed:     4, Skipped:     0, Total:     4, Duration: 11 ms - Calculator.Server.Tests.dll (net7.0)
  Skipped TestOperations("/add/3/5","8") [4 s]
  Skipped TestOperations("/subtract/3/5","-2") [4 s]
  Skipped TestOperations("/multiply/3/5","15") [4 s]
  Skipped TestOperations("/divide/35/5","7") [4 s]

Skipped! - Failed:     0, Passed:     0, Skipped:     4, Total:     4, Duration: 4 s - Calculator.Server.IntegrationTests.dll (net7.0)

Attachments:
  C:\Users\MyUser\codecoverage\samples\Calculator\tests\Calculator.Core.Tests\TestResults\e18df044-bf6a-47a2-a739-e7d0e8a6443f\MyUser_2024-01-01.16_45_46.coverage

As the results show, the coverage of the first run was saved within the Calculator.Server.IntegrationTests\TestResults folder, while the file for the 2nd execution was saved in Calculator.Core.Tests\TestResults

I would expect a solution wide file to be saved in the solution root, or under a TestResults folder in the sln root by default. I can still manually merge with the dotnet-coverage tool, but I was excited at the prospect of not needing the extra steps.

I am running dotnet 7.0.203 on Windows 11. I get the same results with and without the dotnet-coverage tool installed. I am developing in VS Code, but don't believe that is relevant because all of the code samples are from the command line.

[jakubch1]

@aarndt you can try:

dotnet test --collect "Code Coverage;CoverageFileName=merged.coverage" --results-directory D:\jakubch1\codecoverage\samples\Calculator\TestResults

but this will still generate final coverage report under path with random guid. Check: microsoft/vstest#2378

IMO the best way to resolve it is use dotnet-coverage for collection. This will give you also the best performance. Please check two examples:
microsoft/vstest#2378 (comment)
https://github.com/microsoft/codecoverage/blob/main/samples/Calculator/scenarios/scenario21/README.md

[jakubch1]

Added more documentation about using dotnet-coverage in this scenario:
https://github.com/microsoft/codecoverage/blob/main/samples/Calculator/scenarios/scenario25/README.md

[aarndt]

@jakubch1 thank you for the quick response and helpful example. Much appreciated!

This worked well on my windows machine. However, an empty coverage file is generated when I run the command on a M2 ARM64 Macbook (some of our devs work with devcontainers on m1/m2 macs). I attempted this with both x64 and arm64 versions of the dotnet coverage tool. As far as I can tell, using dotnet test --collect "CodeCoverage;Format=Cobertura" then following it up with reportgenerator to combine/convert the output is the only method that works in both scenarios. I suspect this is due to dynamic instrumentation not being supported on macOS arm64, but I must admit my understanding of this is limited.

Edit:
For extra context, our use case is developers being able to easily measure and track code coverage in devcontainer-based projects in VS Code. I would like to create a task that generates a coverage report along with a coverage file that can be consumed by the coverage-gutters extension.

[jakubch1]

Yes, this is not working because macOS arm64 is not supported for dynamic instrumentation. So I recommend to do step back and use:

dotnet test --collect "Code Coverage;CoverageFileName=merged.coverage" --results-directory D:\jakubch1\codecoverage\samples\Calculator\TestResults

and then use some scripting to find final coverage report. dotnet test --collect has some additional callback from Test Platform regarding dlls that will be tested and can by default statically instrument dlls (in envs where dynamic is not supported).

There is also a way to make dotnet-coverage work in this scenario but I don't recommend it as it is quite complicated. Basically you have to build solution before running tests and then specify using --include-files all dlls that should be statically instrumented. So final command would be something like:

dotnet-coverage collect --include-files "**/bin/Debug/*.dll" -f cobertura -o report.cobertura.xml "dotnet test --no-build"

This will make sure that all dlls are statically instrumented before command dotnet test --no-build is executed. This procedure can take time if your solution is big so this is another reason this approach is not recommended.

[jakubch1]

Closing as no work needed here.