[BUG] coverlet.console with vstest.console.exe reporting 0% coverage

[DavidTynewyddBI]

Describe the bug
I'm attempting to get code coverage on a WinUI3 project which includes tests marked with [UITestMethod].

If I ignore the UITestMethods, I can successfully get coverage using dotnet test --collect:"XPlat Code Coverage", however dotnet test doesn't work with UITestMethod, hence using vstest.console.exe.

I'm using coverlet.console, which runs the tests and reports the correct referenced projects but with 0% coverage, along with the message "Hits file not found for module ".

To Reproduce

1. Create an empty winui3 class library with some public classes and methods
2. Add a winui3 test project
3. Reference the class library and call some but not all of the methods
4. Build
5. Run coverlet .\bin\x64\Debug\net8.0-windows10.0.19041.0\win-x64\TestApp1.build.appxrecipe --target "vstest.console.exe" --targetargs "bin\x64\Debug\net8.0-windows10.0.19041.0\win-x64\TestApp1.build.appxrecipe /Logger:trx" --verbosity detailed where TestApp1 is the name of your test project

Expected behavior
Tests to run and code coverage to be produced

Actual behavior
Either reports 0% or 100% while saying the hits file was not found.
e.g.

Terminating app with process ID '2144'.
  Passed TestMethod1 [5 ms]
  Passed TestMethod2 [12 ms]
  Passed TestMethod3 [28 ms]
Results File: <redacted>
Test Run Successful.
Total tests: 3
     Passed: 3
 Total time: 9.4086 Seconds

Calculating coverage result...
Hits file:'<redacted>\AppData\Local\Temp\ClassLibrary1_4dd00cc7-1a92-4f0b-a9ad-b917071ea6ab' not found for module: 'ClassLibrary1'
Hits file:'<redacted>\AppData\Local\Temp\TestApp1_4dd00cc7-1a92-4f0b-a9ad-b917071ea6ab' not found for module: 'TestApp1'
  Generating report '<redacted>\App1\TestApp1\coverage.json'
+---------------+------+--------+--------+
| Module        | Line | Branch | Method |
+---------------+------+--------+--------+
| ClassLibrary1 | 0%   | 100%   | 0%     |
+---------------+------+--------+--------+
| TestApp1      | 0%   | 0%     | 0%     |
+---------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 50%    | 0%     |
+---------+------+--------+--------+

Configuration (please complete the following information):
Please provide more information on your .NET configuration:
* Which coverlet package and version was used? coverlet.console 6.0.4.1+90b21079d4
* Which version of .NET is the code running on? .net8
* What OS and version, and what distro if applicable? Windows 11 26100
* What is the architecture (x64, x86, ARM, ARM64)? x64
* Do you know whether it is specific to that configuration? Unknown

Additional context
I'm aware of the known issue giving 0% coverage, however this is an empty project with essentially empty tests.

❗ Please also read Known Issues

[Bertk]

Thank you for bringing this up.

The warning message is created in Coverage.cs which has this comments.

     // Hits file could be missed mainly for two reason
     // 1) Issue during module Unload()
     // 2) Instrumented module is never loaded or used so we don't have any hit to register and
     //    module tracker is never use

1. Please check the diagnostic log file using parameter --Diag:"log.txt;tracelevel=verbose". Typically assembly load exceptions will give some hints.
2. You should also execute "dotnet publish" after the build step and before test execution using --no-build parameter (see also VSTestIntegration
3. Please check DebugType=portable (see BadImageFormatException .NET Framework 4.7.x, 4.8.x)

[kitUIN]

In .csproj

<PropertyGroup>
  <DebugType>full</DebugType>
</PropertyGroup>

not found for module solved

But also 0%

[kitUIN]

The main issue at present is that VSTest deploys and launches the WinUI 3 project, preventing Coverlet from properly attaching to the application of WinUI 3 for analysis.

[Bertk]

In .csproj

<PropertyGroup>
  <DebugType>full</DebugType>
</PropertyGroup>

not found for module solved

But also 0%

Please check whether assembly load errors occur. I guess your test project targets net8.0 and maybe you have to add additional nuget package reference e.g.

    <PackageReference Include="Microsoft.Bcl.AsyncInterfaces" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.3" />

[kitUIN]

Perhaps it would be better to be a data collector for vstest rather than target

& "D:\Microsoft Visual Studio\2022\Preview\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" bin\x64\Debug\net8.0-windows10.0.19041.0\win-x64\App1.build.appxrecipe /Logger:trx  /Settings:test.runsettings /TestAdapterPath:.

<RunSettings>
	<RunConfiguration>
		<DisableAppDomain>True</DisableAppDomain>
	</RunConfiguration>
	<DataCollectionRunSettings>
		<DataCollectors>
			<DataCollector friendlyName="XPlat Code Coverage" 
						   codebase="coverlet.collector.dll" >
				<Configuration>
					<Format>cobertura</Format>
					<LogLevel>Verbose</LogLevel>
				</Configuration>
			</DataCollector>
		</DataCollectors>
	</DataCollectionRunSettings>

</RunSettings>

<PropertyGroup>
...
  <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
...
</PropertyGroup>

But it's not quite correct on my machine,I'm sure Newtonsoft.Json exists.

数据收集器“XPlat code coverage”消息: [coverlet]Coverlet.Collector.Utilities.CoverletDataCollectorException: CoverletCoverageDataCollector: Failed to get coverage result ---> 
System.IO.FileNotFoundException: 未能加载文件或程序集“Newtonsoft.Json, Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed”或它的某一个依赖项。系统找不到指定的文件。
   在 Coverlet.Core.Coverage.GetCoverageResult()
   在 Coverlet.Collector.DataCollection.CoverageManager.GetCoverageResult() 位置 /_/src/coverlet.collector/DataCollection/CoverageManager.cs:行号 95
   --- 内部异常堆栈跟踪的结尾 ---
   在 Coverlet.Collector.DataCollection.CoverageManager.GetCoverageResult() 位置 /_/src/coverlet.collector/DataCollection/CoverageManager.cs:行号 100
   在 Coverlet.Collector.DataCollection.CoverletCoverageCollector.OnSessionEnd(Object sender, SessionEndEventArgs e) 位置 /_/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs:行号 164。

Now I can't solve this problem

[Bertk]

Please add PackageReference to your test project.

<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />

[kitUIN]

Please add PackageReference to your test project.

<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />

still error😢

[Bertk]

I see. Typically this happens also if an dependency of the assembly cannot be loaded.

Nevertheless the version of Newtonsoft.Json shall be updated because all version before V13.0.1 have vulnerabilities.

[Bertk]

You can find the vulnerable package

dotnet list package --vulnerable --include-transitive