Skip to content

Coverlet Collectors intermittently fail to collect coverage results #893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
orbus-dkreivenas opened this issue Jul 9, 2020 · 7 comments
Closed

Coverlet Collectors intermittently fail to collect coverage results #893

orbus-dkreivenas opened this issue Jul 9, 2020 · 7 comments
Labels
as-designed Expected behaviour Known Issue It's a know issue waiting for customer Waiting for customer action

Comments

@orbus-dkreivenas
Copy link

orbus-dkreivenas commented Jul 9, 2020 •
edited
Loading

Hello, we are experiencing a problem using the collectors. Our package version are the following:

  • coverlet.collectors 1.3.0
  • Microsoft.NET.Test.Sdk 16.6.1
  • Tests projects target netcoreapp2.2
  • dotnet sdk 3.1.301
  • Microsoft Test Execution Command Line Tool Version 16.6.0 (displayed in the test logs)

We run the tests by running dotnet test {CS Project} --no-build --no-restore --logger trx --collect "XPlat Code Coverage".

The error we receive is:

Data collector 'XPlat code coverage' message: [coverlet]Coverlet.Collector.Utilities.CoverletDataCollectorException: CoverletCoverageDataCollector: Failed to get coverage result
 ---> System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.InternalRead(Int32 numBytes)
   at Coverlet.Core.Coverage.CalculateCoverage() in /_/src/coverlet.core/Coverage.cs:line 367
   at Coverlet.Core.Coverage.GetCoverageResult() in /_/src/coverlet.core/Coverage.cs:line 141
   at Coverlet.Collector.DataCollection.CoverageWrapper.GetCoverageResult(Coverage coverage) in /_/src/coverlet.collector/DataCollection/CoverageWrapper.cs:line 44
   at Coverlet.Collector.DataCollection.CoverageManager.GetCoverageResult() in /_/src/coverlet.collector/DataCollection/CoverageManager.cs:line 93
   --- End of inner exception stack trace ---
   at Coverlet.Collector.DataCollection.CoverageManager.GetCoverageResult() in /_/src/coverlet.collector/DataCollection/CoverageManager.cs:line 98
   at Coverlet.Collector.DataCollection.CoverletCoverageCollector.OnSessionEnd(Object sender, SessionEndEventArgs e) in /_/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs:line 160.

The test projects are being run one by one. This does not happen all the time, hence our coverage is inconsistent.

Could it be a problem that our projects consist of net472 and netcoreapp2.2 ?
@MarcoRossignoli
Copy link
Collaborator

net472 is not supported with collectors #705 (comment)
Btw that error is strange on collectors, can you enable logging and attach(warning: take a look at logs before send becasue it contains libs names and could expose secrets)?
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#collectors-integration VSTest logging
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#enable-injected-tracker-log our internal tracker logging

@MarcoRossignoli MarcoRossignoli added the waiting for customer Waiting for customer action label Jul 10, 2020
@orbus-dkreivenas
Copy link
Author

I have tried running the dotnet test {CS Project} --no-build --no-restore --logger trx --collect "XPlat Code Coverage" for a net472 and the coverage result was collected. It is a SDK style project targeting the net472 framework.

Here are the logs:
log.datacollector.20-07-10_14-52-23_42566_4.zip

@MarcoRossignoli
Copy link
Collaborator

MarcoRossignoli commented Jul 11, 2020 •
edited
Loading

Coverlet collectors atm are no supported in .NET Framework, only .NET Core as we reported in guide https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md#coverlet-integration-with-vstest-aka-visual-studio-test-platform

TL;DR;

As you can see in log log.host.20-07-10_14-52-30_81650_4.txt there are a lot of fails in type resolution

TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.218, 12692512464121, testhost.x86.exe, TestPluginCache.GetExtensionPaths: Added unfilterableExtensionPaths: 
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512501003, testhost.x86.exe, AssemblyResolver.AddSearchDirectories: Adding more searchDirectories C:\Users\oabuild\.nuget\packages\coverlet.collector\1.3.0\build\netstandard1.0
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512516883, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512518579, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512519143, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512522017, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Failed to load assembly.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512525764, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512526336, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512526819, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512527321, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolved from cache.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512530276, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512531504, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512532049, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512532547, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolved from cache.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512535500, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512536098, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512536579, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512537036, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolved from cache.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512539998, testhost.x86.exe, CurrentDomain_AssemblyResolve: Resolving assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Verbose: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512540564, testhost.x86.exe, CurrentDomainAssemblyResolve: Failed to resolve assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
TpTrace Information: 0 : 8980, 5, 2020/07/10, 14:52:32.233, 12692512541084, testhost.x86.exe, AssemblyResolver.OnResolve: System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly.

This means that in-process coverlet collector is not loaded in test process because coverlet lib is compiled for netcoreapp and not for netxxx app. When in-process collector is not loaded "sometimes"(randomic related to vstest platform behaviour) process is killed before coverlet flush hits file to disk, because this collector cannot tell to vstest plat to wait. In these cases(randomic) you hit know issue https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test

So your coverage is inconsistent because sometimes process is killed sometimes not. But it's out of control of coverlet.

@MarcoRossignoli MarcoRossignoli added the Known Issue It's a know issue label Jul 11, 2020
@orbus-dkreivenas
Copy link
Author

So is it possible to resolve this using coverlet.msbuild package for the .NET Framework or the only way is to upgrade to .NET Core ?

@MarcoRossignoli
Copy link
Collaborator

MarcoRossignoli commented Jul 13, 2020 •
edited
Loading

So is it possible to resolve this using coverlet.msbuild package for the .NET Framework or the only way is to upgrade to .NET Core ?

You can use msbuild but you could incur in same issue https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test
So atm there is no way(apart convert to .NET Core) 😞 btw I'm in contact with MS guys to fix it cc: @nohwnd

@MarcoRossignoli MarcoRossignoli added the as-designed Expected behaviour label Jul 13, 2020
@orbus-dkreivenas
Copy link
Author

Thanks for the help. I will keep track on this as we do have projects we can't move to .NET Core. So closing this.

@MarcoRossignoli
Copy link
Collaborator

Keep an eye on this issue #705 (comment) where we track that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
as-designed Expected behaviour Known Issue It's a know issue waiting for customer Waiting for customer action
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MarcoRossignoli @orbus-dkreivenas