🧪 [MTP] AwesomeAssertions throws different NUnit.Framework.AssertionException than expected at runtime

[cbersch]

I was trying to run our existing tests which use NUnit and AwesomeAssertions with MTP. However, I got an error of the type

  NUnit.Framework.AssertionException : Expected a <NUnit.Framework.AssertionException> to be thrown, but found <NUnit.Framework.AssertionException>:
NUnit.Framework.AssertionException: Expected string to be <null>, but found "foo".
   at AwesomeAssertions.Execution.LateBoundTestFramework.Throw(String message)
[...]

The cause of this error is, that the exception we throw is from a different Assembly (the values of Exception.GetType().Assembly.m_assembly differ). We get it by scanning AppDomain.CurrentDomain.GetAssemblies(), which accesses the default AssemblyLoadContext.

The problem appears with NUnit3TestAdapter v 6.0.0-alpha.100, but not with 5.2.0.
And only, if the NUnit.Framework.FixtureLifeCycle is specified in the project itself.

I know, a lot of 'ifs', but I'm stuck looking for the cause, or what we could do on part of AwesomeAssertions to fix this.

Repro:
MyProject.csproj:

<Project Sdk="Microsoft.NET.Sdk">
	<PropertyGroup>
		<TargetFramework>net10.0</TargetFramework>
		<ImplicitUsings>enable</ImplicitUsings>
		<EnableNUnitRunner>true</EnableNUnitRunner>
		<OutputType>Exe</OutputType>
	</PropertyGroup>

	<ItemGroup>
		<PackageReference Include="AwesomeAssertions" Version="9.2.1" />
		<PackageReference Include="NUnit" Version="4.4.0" />
		<PackageReference Include="NUnit3TestAdapter" Version="6.0.0-alpha.100" />
	</ItemGroup>

	<ItemGroup>
		<AssemblyAttribute Include="NUnit.Framework.FixtureLifeCycle">
		<_Parameter1>NUnit.Framework.LifeCycle.InstancePerTestCase</_Parameter1>
		<_Parameter1_IsLiteral>true</_Parameter1_IsLiteral>
		</AssemblyAttribute>
	</ItemGroup>
</Project>

Test.cs:

using AwesomeAssertions;
using NUnit.Framework;

public sealed class ExceptionTests
{
    [Test]
    public void TestMethod_AwesomeAssertions()
    {
        Action act = () => "foo".Should().BeNull();

        act.Should().Throw<AssertionException>();
    }
}

Either commenting the AssemblyAttribute or changing the adapter version to 5.2.0 fixes the problem.

Any clue, which changes in the adapter may cause this?

[OsirisTerje]

Thanks @cbersch !

That was a bit surprising, because nothing is changed in that direction. But obviously, something must have. I'll try to have a look tomorrow on this. We got to have this fixed before release.

PS: If you have a bit of time, could you create a small repro project PR - with the code you have above - in the repo https://github.com/nunit/nunit3-vs-adapter.issues, create a folder named Issue1348 and place the code there :-)

[cbersch]

PS: If you have a bit of time, could you create a small repro project PR - with the code you have above - in the repo https://github.com/nunit/nunit3-vs-adapter.issues, create a folder named Issue1348 and place the code there :-)

Done

[OsirisTerje]

Adapter version 5.2.0 uses MTP version 1
Adapter version 6.0.0 uses MTP version 2

I have added a runsettings file and looked at the output there, and no indication that NUnit itself does anything here. It seems this must be related to some changes in how MTP version 2 runs the test.

@Youssef1313 Can you comment on this ?

@manfred-brands Does the fact this is affected by the lifecycle makes any sense to you?

[Youssef1313]

In MTP v2, we disable app domain by default.
Can you see if you are able to repro with 5.2.0 when app domain is disabled (edit: enabled)?
Also worth trying the same with VSTest when app domain is disabled as well.

[cbersch]

In MTP v2, we disable app domain by default. Can you see if you are able to repro with 5.2.0 when app domain is disabled?

@Youssef1313 How can I disable that?

[OsirisTerje]

<RunSettings>
  <RunConfiguration>
    <DisableAppDomain>true</DisableAppDomain>
  </RunConfiguration>
</RunSettings>

[Youssef1313]

Actually just noticed the repro is .NET 10.
@OsirisTerje I hope NUnit doesn't make any use of DisableAppDomain value when running under .NET (Core)?

[OsirisTerje]

In .net core / .net X, we don't.

I am testing using .net 9 btw.

[OsirisTerje]

Btw, the error seems to only appear when the Lifecycle attribute is on assembly level, if it is on fixture level it works as it should.

[cbersch]

Btw, the error seems to only appear when the Lifecycle attribute is on assembly level, if it is on fixture level it works as it should.

And it appears only when the attribute is set in the csproj file. When adding it with [assembly:...] to a .cs file it works as well.

And I couldn't reproduce it with MSTest. Tried v 4.0.1 and defined the ClassCleanupExecutionAttribute in the csproj file, which also worked fine.

[cbersch]

I see a similar error in code which involves static constructors for scanning code for interface implementations. The error here is

System.TypeInitializationException : The type initializer for 'MyNamespace.ArrayConvert`2' threw an exception.
    ----> System.InvalidCastException : [A]MyNamespace.ArrayConverterAttribute cannot be cast to [B]MyNamespace.ArrayConverterAttribute.
   Type A originates from 'MyNamespace, Version=2025.11.0.0, Culture=neutral, PublicKeyToken=null'
   in the context 'Default' at location '...\Release\Assemblies\MyNamespace.Test\MyNamespace.dll'.
   Type B originates from 'MyNamespace, Version=2025.11.0.0, Culture=neutral, PublicKeyToken=null'
   in the context '"" NUnit.Engine.Internal.TestAssemblyLoadContext #1' at location '...\Release\Assemblies\MyNamespace.Test\MyNamespace.dll'.

I'll try to see if I can create a repro for this situation as well.