Unable to get coverage when running tests with Microsoft.AspNetCore.Mvc.Testing

[NatMarchand]

Hello there,
I have an issue with a simple repro solution:
Three projects :

• WebApp (netcoreapp3.1 with aspnetcore, very simple api)
• Business (netstandard2.0 with only one static class/method)
• Tests (netcoreapp3.1 with xunit and two tests)

The first test instantiate my api controller executes the method and assert on the result.
The second one do almost the same but by instantiating the whole aspnetcore pipeline via the WebApplicationFactory.

The version are as follows:

coverlet.msbuild Version="2.9.0"
Microsoft.AspNetCore.Mvc.Testing Version="3.1.5"
Microsoft.NET.Test.Sdk Version="16.6.1"
xunit Version="2.4.1"
xunit.runner.visualstudio Version="2.4.2"

public class UnitTest1
{
    [Fact]
    public void Test1()
    {
        var sut = new DataController();
        var actionResult = sut.Get() as OkObjectResult;
        Assert.NotNull(actionResult);
        Assert.StartsWith("H", (string)actionResult.Value);
        Assert.EndsWith("!", (string)actionResult.Value);
        var t = typeof(WebApplicationFactory<Startup>);
    }

    //[Fact(Skip = "meh")]
    [Fact]
    public async Task Test2()
    {
        var webApplicationFactory = new WebApplicationFactory<Startup>()
            .WithWebHostBuilder(builder =>
            {
                builder.UseEnvironment("Development");
            });
        using var client = webApplicationFactory.CreateClient();
        var r = await client.GetAsync("data");
        var c = await r.Content.ReadAsStringAsync();
        Assert.StartsWith("H", c);
        Assert.EndsWith("!", c);
        r.EnsureSuccessStatusCode();
    }
}

When I run as is I got this coverage :

dotnet test .\Tests\Tests.csproj -p:CollectCoverage=true -p:CoverletOutputFormat="json%2copencover%2ccobertura"

+----------+--------+--------+--------+
| Module   | Line   | Branch | Method |
+----------+--------+--------+--------+
| Business | 0%     | 0%     | 0%     |
+----------+--------+--------+--------+
| WebApp   | 86,36% | 100%   | 80%    |
+----------+--------+--------+--------+

When I skip Test2 I got this coverage:

dotnet test .\Tests\Tests.csproj -p:CollectCoverage=true -:CoverletOutputFormat="json%2copencover%2ccobertura"

+----------+--------+--------+--------+
| Module   | Line   | Branch | Method |
+----------+--------+--------+--------+
| Business | 71,42% | 50%    | 100%   |
+----------+--------+--------+--------+
| WebApp   | 18,18% | 100%   | 20%    |
+----------+--------+--------+--------+

It seems that hosting with WebApplicationFactory makes the coverage unavailable.
I can provide the whole solution if needed.

My env :

> dotnet --version
3.1.400-preview-015178

> [System.Environment]::OSVersion.Version
Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      19042  0

[MarcoRossignoli]

I can provide the whole solution if needed.

Could be related to known issue https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test
Can you try with collectors integration? https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md

It it doesn't work please can you provide the solution so I can test on my local?

[NatMarchand]

After checking, it does work with collector. Unfortunately, I picked the msbuild way because I needed the mergewith. Do you have any update about supporting MergeWith with collector ?

[MarcoRossignoli]

We're working on some solutions, but I don't have eta at the moment, btw you can use a trick used by one of our user #225 (comment)

Also check if your coverage report generator can merge files like reportgenerator https://www.palmmedia.de/OpenSource/ReportGenerator

[NatMarchand]

Yep, I’ve already seen it. Might try tomorrow but not a huge fan of this workaround as it complicates my pipeline and might not integrate well with the dotnet test task I use (working on azure devops).
For the report, I know that Azure Devops merges them but I’m not sure for Sonarqube.
Thx a lot by the way.
Would you consider opening a task in the project board in order to track progress ? Could be great 😉

[MarcoRossignoli]

Might try tomorrow but not a huge fan of this workaround as it complicates my pipeline and might not integrate well with the dotnet test task I use (working on azure devops).

I undestand you 😞

Would you consider opening a task in the project board in order to track progress ? Could be great 😉

You can follow our changelog and roadmap(expected, btw could change), atm we don't use the board because coverlet is OSS and it's not easy do long term planning

https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Changelog.md
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Roadmap.md