No coverage reported when using RavenDb.TestDriver package

[jmezach]

I'm having an issue using Coverlet when I use RavenDb.TestDriver in my tests. As soon as I add a call to its GetDocumentStore() function I'm getting zero coverage. Here are the steps to reproduce the issue:

• dotnet new xunit
• dotnet add package coverlet.msbuild
• dotnet add package RavenDb.TestDriver
• Edit UnitTest1.cs and let the UnitTest1 class inherit from RavenTestDriver
• Add a call to GetDocumentStore() to Test1 ie: var store = GetDocumentStore()
• dotnet test /p:CollectCoverage=true

Expect behaviour:

➜ dotnet test /p:CollectCoverage=true
Build started, please wait...
Build completed.

Test run for /Users/jmezach/Projects/demos/CoverletRavenDbTestDriverIssue/bin/Debug/netcoreapp2.2/CoverletRavenDbTestDriverIssue.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.9505 Seconds

Calculating coverage result...
  Generating report '/Users/jmezach/Projects/demos/CoverletRavenDbTestDriverIssue/coverage.json'

+--------------------------------------------------+--------+--------+--------+
| Module                                           | Line   | Branch | Method |
+--------------------------------------------------+--------+--------+--------+
| xunit.runner.reporters.netcoreapp10              | 1.1%   | 0.5%   | 5.1%   |
+--------------------------------------------------+--------+--------+--------+
| xunit.runner.visualstudio.dotnetcore.testadapter | 45.8%  | 35.8%  | 47.8%  |
+--------------------------------------------------+--------+--------+--------+
| xunit.runner.utility.netcoreapp10                | 15.7%  | 9.1%   | 21%    |
+--------------------------------------------------+--------+--------+--------+

+---------+-------------------+-------------------+-------------------+
|         | Line              | Branch            | Method            |
+---------+-------------------+-------------------+-------------------+
| Total   | 23.8%             | 17.9%             | 26.6%             |
+---------+-------------------+-------------------+-------------------+
| Average | 7.93333333333333% | 5.96666666666667% | 8.86666666666667% |
+---------+-------------------+-------------------+-------------------+

Actual behavior:

➜ dotnet test /p:CollectCoverage=true
Build started, please wait...
Build completed.

Test run for /Users/jmezach/Projects/demos/CoverletRavenDbTestDriverIssue/bin/Debug/netcoreapp2.2/CoverletRavenDbTestDriverIssue.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 7.3330 Seconds

Calculating coverage result...
  Generating report '/Users/jmezach/Projects/demos/CoverletRavenDbTestDriverIssue/coverage.json'

+--------------------------------------------------+--------+--------+--------+
| Module                                           | Line   | Branch | Method |
+--------------------------------------------------+--------+--------+--------+
| xunit.runner.reporters.netcoreapp10              | 0%     | 0%     | 0%     |
+--------------------------------------------------+--------+--------+--------+
| xunit.runner.visualstudio.dotnetcore.testadapter | 0%     | 0%     | 0%     |
+--------------------------------------------------+--------+--------+--------+
| xunit.runner.utility.netcoreapp10                | 0%     | 0%     | 0%     |
+--------------------------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 0%     | 0%     | 0%     |
+---------+--------+--------+--------+
| Average | 0%     | 0%     | 0%     |
+---------+--------+--------+--------+

I'm not sure if there's a way to get more diagnostic output from Coverlet to figure out what's going on, but please let me know if there is.

[tonerdo]

#341 is currently adding support for diagnostic output in Coverlet which should give us more insight into the problem. We have a couple of suspicions for this problem and efforts like #329 are underway to figure out how best to fix it

[jmezach]

Okay, looking forward to #341 getting merged ;). If there's anything to try out already, let me know, I would be willing to test drive.

[jmezach]

FYI, I checked out the code from #341 and ran it on my project using coverlet.console and while I am seeing some more messages, I still don't see any reason for the coverage being zero.

[MarcoRossignoli]

@jmezach your issue could not be logged yet, you should try to add logging here(also a simple Console.WriteLine) and understand if hits file is missing(related to #329)

[jmezach]

@MarcoRossignoli I added the logging in my local build and I now get the following output:

File not instrumented, or nothing in it called: /var/folders/z4/dxl_s1v53rx11b9ypgzy97mr0000gn/T/Augurk.Test_f81ad29f-7375-449a-9a36-3562e4f7b1ce
File not instrumented, or nothing in it called: /var/folders/z4/dxl_s1v53rx11b9ypgzy97mr0000gn/T/Augurk_f81ad29f-7375-449a-9a36-3562e4f7b1ce
File not instrumented, or nothing in it called: /var/folders/z4/dxl_s1v53rx11b9ypgzy97mr0000gn/T/Augurk.Entities_f81ad29f-7375-449a-9a36-3562e4f7b1ce

I'm pretty sure my code is being called, so either the assemblies aren't being properly instrumented or something else is going on. I'll see if I can get a handle on this ;).

[MarcoRossignoli]

@tonerdo could be related to #329 also for Linux?

/cc @sharwell @ViktorHofer

[tonerdo]

Can say for sure. Possibly the assembly wasn't instrumented or it was instrumented and the temp file couldn't be created. @jmezach what version of Coverlet are you using?

[jmezach]

@tonerdo I'm currently using locally build bits based on the sources from #341 in order to figure out what's going on, but I was using 2.5.1 when I initially reported this issue.

I've managed to add some additional logging inside of the instrumented assembly by tweaking the ModuleTrackerTemplate. I can see that the RecordHit method there is being called while the tests are executing. But I also added a Console.WriteLine at the top of UnloadModule which I do not see in my output, so it looks like the event isn't triggered.

After reading #110 I'm fairly sure I'm hitting the same problem where VSTest is already shutting down the test process before the results file is generated. Guess I'll have to wait for #329 to land. Any updates on the progress there?

[embarrassedmilk]

@jmezach did you work around this problem by any chance? I'm using 2.6.3 atm and issue still seems to be there.

What I noticed is that when I set TargetFramework to e.g. net472 and use flag /p:Exclude="[Raven.*]*%2c[xunit.*]*%2c[Sparrow*]*", it works. But not on netcoreapp

[MarcoRossignoli]

Solved with usage of in process collectors https://github.com/tonerdo/coverlet#requirements https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md
NB at the moment there is a know issue on vstest side so to use updated version of collectors(> 1.0.0 you need to provide custom runsettings #576