Incompatibility with xunit and netfx

[tasadar2]

Problem

When working with a netfx project that uses xunit, dotnet test with coverlet fails with the following error:

dotnet test coverlet.diag.unittests/coverlet.diag.unittests.csproj /p:CollectCoverage=true

Build started, please wait...
Build completed.

Test run for coverlet.diag.unittests\bin\Debug\net461\coverlet.diag.unittests.dll(.NETFramework,Version=v4.6.1)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.26] coverlet.diag.unittests: Catastrophic failure: System.IO.FileLoadException: Could not load file or assembly 'xunit.runner.utility.net452, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. Strong name signature could not be verified.  The assembly may have been tampered with, or it was delay signed but not fully signed with the correct private key. (Exception from HRESULT: 0x80131045)
File name: 'xunit.runner.utility.net452, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   at Xunit.AppDomainManager_AppDomain.CreateObject[TObject](AssemblyName assemblyName, String typeName, Object[] args)
   at Xunit.DiaSessionWrapper..ctor(String assemblyFilename)
   at Xunit.VisualStudioSourceInformationProvider..ctor(String assemblyFileName)
   at Xunit.XunitFrontController..ctor(AppDomainSupport appDomainSupport, String assemblyFileName, String configFileName, Boolean shadowCopy, String shadowCopyFolder, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

No test is available in coverlet.diag.unittests\bin\Debug\net461\coverlet.diag.unittests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Test Run Failed.
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

Environment

• winver: Windows 10 Version 1607 (OS Build 14393.2724)
• dotnet --version: 2.1.503

Troubleshooting

• I initially encountered this with net471, but seems to be consistent with net461 and up.
• This started happening when updating coverlet.msbuild beyond 2.4.0.
• It would seem that coverlet is injecting into some xunit assemblies.

Replication

I created a sample repo which fails when tests are run with coverage

1. Clone https://github.com/tasadar2/coverlet.diag

   git clone https://github.com/tasadar2/coverlet.diag coverlet.diag
   cd coverlet.diag

2. Run tests with coverage

   dotnet test coverlet.diag.unittests/coverlet.diag.unittests.csproj /p:CollectCoverage=true

Workarounds

• Exclude the xunit runner, or all xunit assemblies during test(I would think it would do this anyway since there are no pdbs in the nuget package)

  dotnet test coverlet.diag.unittests/coverlet.diag.unittests.csproj /p:CollectCoverage=true /p:Exclude="[xunit.*]*"

• Downgrade to 2.4.0

  dotnet add coverlet.diag.unittests\coverlet.diag.unittests.csproj package coverlet.msbuild --version 2.4.0

[eMarkM]

Just ran into this myself. I have .Net Core project using net462 with xunit 2.4.1 as my test framework. I tried both workarounds, but still get the same exact error. If I take out ref to xunit.runner.visualstudio, it will produce a coverage file, but then says there are no test in the project and the code coverage file is all zeros.

[tasadar2]

@eMarkM Not sure if it helps, but when I first downgraded, I ran into the issue, but something was still cached. I had to removed the obj and bin directories from my projects to force a completely fresh run.

[eMarkM]

yeah, I got it to work finally with /p:Exclude="[xunit.]" flag. Thanks.