Coverlet Collectors intermittently fail to collect coverage results

[orbus-dkreivenas]

Hello, we are experiencing a problem using the collectors. Our package version are the following:

• coverlet.collectors 1.3.0
• Microsoft.NET.Test.Sdk 16.6.1
• Tests projects target netcoreapp2.2
• dotnet sdk 3.1.301
• Microsoft Test Execution Command Line Tool Version 16.6.0 (displayed in the test logs)

We run the tests by running dotnet test {CS Project} --no-build --no-restore --logger trx --collect "XPlat Code Coverage".

The error we receive is:

Data collector 'XPlat code coverage' message: [coverlet]Coverlet.Collector.Utilities.CoverletDataCollectorException: CoverletCoverageDataCollector: Failed to get coverage result
 ---> System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.InternalRead(Int32 numBytes)
   at Coverlet.Core.Coverage.CalculateCoverage() in /_/src/coverlet.core/Coverage.cs:line 367
   at Coverlet.Core.Coverage.GetCoverageResult() in /_/src/coverlet.core/Coverage.cs:line 141
   at Coverlet.Collector.DataCollection.CoverageWrapper.GetCoverageResult(Coverage coverage) in /_/src/coverlet.collector/DataCollection/CoverageWrapper.cs:line 44
   at Coverlet.Collector.DataCollection.CoverageManager.GetCoverageResult() in /_/src/coverlet.collector/DataCollection/CoverageManager.cs:line 93
   --- End of inner exception stack trace ---
   at Coverlet.Collector.DataCollection.CoverageManager.GetCoverageResult() in /_/src/coverlet.collector/DataCollection/CoverageManager.cs:line 98
   at Coverlet.Collector.DataCollection.CoverletCoverageCollector.OnSessionEnd(Object sender, SessionEndEventArgs e) in /_/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs:line 160.

The test projects are being run one by one. This does not happen all the time, hence our coverage is inconsistent.

Could it be a problem that our projects consist of net472 and netcoreapp2.2 ?

[MarcoRossignoli]

net472 is not supported with collectors #705 (comment)
Btw that error is strange on collectors, can you enable logging and attach(warning: take a look at logs before send becasue it contains libs names and could expose secrets)?
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#collectors-integration VSTest logging
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#enable-injected-tracker-log our internal tracker logging

[orbus-dkreivenas]

I have tried running the dotnet test {CS Project} --no-build --no-restore --logger trx --collect "XPlat Code Coverage" for a net472 and the coverage result was collected. It is a SDK style project targeting the net472 framework.

Here are the logs:
log.datacollector.20-07-10_14-52-23_42566_4.zip

[MarcoRossignoli]

Coverlet collectors atm are no supported in .NET Framework, only .NET Core as we reported in guide https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md#coverlet-integration-with-vstest-aka-visual-studio-test-platform

TL;DR;

As you can see in log log.host.20-07-10_14-52-30_81650_4.txt there are a lot of fails in type resolution

TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.218, 12692512464121, testhost.x86.exe, TestPluginCache.GetExtensionPaths: Added unfilterableExtensionPaths: 
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512501003, testhost.x86.exe, AssemblyResolver.AddSearchDirectories: Adding more searchDirectories C:\Users\oabuild\.nuget\packages\coverlet.collector\1.3.0\build\netstandard1.0
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512516883, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512518579, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512519143, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512522017, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Failed to load assembly.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512525764, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512526336, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512526819, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512527321, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolved from cache.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512530276, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512531504, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512532049, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512532547, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolved from cache.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512535500, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512536098, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512536579, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512537036, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolved from cache.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512539998, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512540564, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512541084, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.

This means that in-process coverlet collector is not loaded in test process because coverlet lib is compiled for netcoreapp and not for netxxx app. When in-process collector is not loaded "sometimes"(randomic related to vstest platform behaviour) process is killed before coverlet flush hits file to disk, because this collector cannot tell to vstest plat to wait. In these cases(randomic) you hit know issue https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test

So your coverage is inconsistent because sometimes process is killed sometimes not. But it's out of control of coverlet.

[orbus-dkreivenas]

So is it possible to resolve this using coverlet.msbuild package for the .NET Framework or the only way is to upgrade to .NET Core ?

[MarcoRossignoli]

So is it possible to resolve this using coverlet.msbuild package for the .NET Framework or the only way is to upgrade to .NET Core ?

You can use msbuild but you could incur in same issue https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test
So atm there is no way(apart convert to .NET Core) 😞 btw I'm in contact with MS guys to fix it cc: @nohwnd

[orbus-dkreivenas]

Thanks for the help. I will keep track on this as we do have projects we can't move to .NET Core. So closing this.

[MarcoRossignoli]

Keep an eye on this issue #705 (comment) where we track that issue.