.Net 7 preview 7 is broken on linux arm64 via QEMU

[MichalPetryka]

Describe the bug

Changing

./dotnet-install.sh -c 6.0

to

./dotnet-install.sh -c 6.0
./dotnet-install.sh -c 7.0

made my test script error out on alpine arm64. Looks like something broke in the SDK for that platform in .Net 7, since it works fine with just .Net 6.

To Reproduce

This error seems to happen on dotnet test.

Exceptions (if any)

1st run

  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018: The "ResolveAppHosts" task failed unexpectedly. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018: System.ArgumentNullException: Value cannot be null. (Parameter 'source') [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at System.Linq.Enumerable.Where[TSource](IEnumerable`1 source, Func`2 predicate) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at Microsoft.NET.Build.Tasks.ResolveAppHosts.ExecuteCore() [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018: The "ResolveAppHosts" task failed unexpectedly. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018: System.ArgumentNullException: Value cannot be null. (Parameter 'source') [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at System.Linq.Enumerable.Where[TSource](IEnumerable`1 source, Func`2 predicate) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at Microsoft.NET.Build.Tasks.ResolveAppHosts.ExecuteCore() [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(134,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]

2nd run:

  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5034,7): error MSB4018: The "CallTarget" task failed unexpectedly. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5034,7): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5034,7): error MSB4018:    at InvokeStub_CallTarget.get_TargetOutputs(Object, Object, IntPtr*) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5034,7): error MSB4018:    at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5034,7): error MSB4028: The "CallTarget" task's outputs could not be retrieved from the "TargetOutputs" parameter. Object reference not set to an instance of an object. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5477,7): error MSB4018: The "FindUnderPath" task failed unexpectedly. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5477,7): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5477,7): error MSB4018:    at InvokeStub_FindUnderPath.get_InPath(Object, Object, IntPtr*) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5477,7): error MSB4018:    at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Microsoft.Common.CurrentVersion.targets(5477,7): error MSB4028: The "FindUnderPath" task's outputs could not be retrieved from the "InPath" parameter. Object reference not set to an instance of an object. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]

Further technical details

   Version:   7.0.100-preview.7.22377.5
   Commit:    ba310d9309
  Runtime Environment:
   OS Name:     alpine
   OS Version:  3.16
   OS Platform: Linux
   RID:         alpine.3.16-arm64
   Base Path:   /root/.dotnet/sdk/7.0.100-preview.7.22377.5/
  Host:
    Version:      7.0.0-preview.7.22375.6
    Architecture: arm64
    Commit:       eecb028078
  .NET SDKs installed:
    6.0.400 [/root/.dotnet/sdk]
    7.0.100-preview.7.22377.5 [/root/.dotnet/sdk]
  .NET runtimes installed:
    Microsoft.AspNetCore.App 6.0.8 [/root/.dotnet/shared/Microsoft.AspNetCore.App]
    Microsoft.AspNetCore.App 7.0.0-preview.7.22376.6 [/root/.dotnet/shared/Microsoft.AspNetCore.App]
    Microsoft.NETCore.App 6.0.8 [/root/.dotnet/shared/Microsoft.NETCore.App]
    Microsoft.NETCore.App 7.0.0-preview.7.22375.6 [/root/.dotnet/shared/Microsoft.NETCore.App]
  Other architectures found:
    None
  Environment variables:
    DOTNET_ROOT       [/root/.dotnet]
  global.json file:
    Not found
  Learn more:
    https://aka.ms/dotnet/info
  Download .NET:
    https://aka.ms/dotnet/download

[danmoseley]

Does it work with only 7 installed?

[MichalPetryka]

With just 7 installed dotnet new xunit gave:

  Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at InvokeStub_Assembly.get_Location(Object, Object, IntPtr*)
     at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
     --- End of inner exception stack trace ---
     at Microsoft.Build.Shared.BuildEnvironmentHelper.get_Instance()
     at Microsoft.Build.CommandLine.MSBuildApp.ConstructArrayArg(String[] args)
     at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
     at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
  Restore failed.
  Post action failed.
  Manual instructions: Run 'dotnet restore'

And dotnet test gave:

  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4018: The "ProcessFrameworkReferences" task failed unexpectedly. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4018:    at InvokeStub_ProcessFrameworkReferences.get_UnavailableRuntimePacks(Object, Object, IntPtr*) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4018:    at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4028: The "ProcessFrameworkReferences" task's outputs could not be retrieved from the "UnavailableRuntimePacks" parameter. Object reference not set to an instance of an object. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4018: The "ProcessFrameworkReferences" task failed unexpectedly. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4018:    at InvokeStub_ProcessFrameworkReferences.get_UnavailableRuntimePacks(Object, Object, IntPtr*) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4018:    at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]
  /root/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(124,7): error MSB4028: The "ProcessFrameworkReferences" task's outputs could not be retrieved from the "UnavailableRuntimePacks" parameter. Object reference not set to an instance of an object. [/home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium/BetterCallSodium.csproj]

[MichalPetryka]

Disabling W^X via export DOTNET_EnableWriteXorExecute=0 makes those errors disappear but then both debian arm64 (which works fine without it tested again, it crashes with the same errors on .net 7.0, works fine with 6.0) and alpine arm64 just exit the process with no message:

  Starting test execution, please wait...
  A total of 1 test files matched the specified pattern.
  /home/runner/work/BetterCallSodium/BetterCallSodium/BetterCallSodium.Tests/bin/Debug/net7.0/BetterCallSodium.Tests.dll

For reference, the environment is a Docker QEMU container running on GitHub actions.
@janvorli

[janvorli]

@MichalPetryka can you please share steps to create a docker qemu container so that I can debug the problem?
Also, I am a bit confused by your comments above. So when you disable the W^X, you've said that the error disappear, but then you say that the arm64 alpine just exits. So do I read it correctly that it doesn't work whether W^X is enabled or disabled and just the issue effect is different?
And do I understand it correctly that the debian behaves exactly the same way as Alpine, so the issue is not Alpine specific?

[MichalPetryka]

@MichalPetryka can you please share steps to create a docker qemu container so that I can debug the problem? Also, I am a bit confused by your comments above. So when you disable the W^X, you've said that the error disappear, but then you say that the arm64 alpine just exits. So do I read it correctly that it doesn't work whether W^X is enabled or disabled and just the issue effect is different? And do I understand it correctly that the debian behaves exactly the same way as Alpine, so the issue is not Alpine specific?

That's right, it doesn't work with W^X both enabled and disabled, with it enabled it throws random errors that look like GC holes and with it disabled there's no message. And yeah, the issue is not alpine specific (which I've originally thought).

For the container I use the run-on-arch action, you can find my full script here (I've removes all oses other than debian there to avoid actions getting cancelled due to other jobs failing).

[janvorli]

This is basically a dup of dotnet/runtime#13648, also discussed in dotnet/runtime#72227. I can repro it thanks to your instructions, but debugging in qemu emulated container is tricky. I am still trying.

[MichalPetryka]

Worth noting is that .Net 6 works fine with arm64 and s390x. (arm32 fails for both 6 and 7 but with different issues)

[janvorli]

@MichalPetryka thank you for the info, it is important to know it is a regression.

[Syzuna]

Not sure if this is related but I tried building a .NET 7 container image with QEMU for my ARM64 server with github actions (sadly there still are no native runners) and I ran into the following issue (.NET 6 works perfectly)

#21 [publish 1/1] RUN dotnet publish -c Release --no-build -o /app/publish
#21 11.04 MSBuild version 17.4.0-preview-22368-02+c8492483a for .NET
#21 63.59 /usr/share/dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(1103,5): error MSB4018: The "ValidateExecutableReferences" task failed unexpectedly. [/src/EDAPI/SyzunaPrograms.ED.EDAPI/SyzunaPrograms.ED.EDAPI.csproj]
#21 63.59 /usr/share/dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(1103,5): error MSB4018: System.InvalidOperationException: Sequence contains no elements [/src/EDAPI/SyzunaPrograms.ED.EDAPI/SyzunaPrograms.ED.EDAPI.csproj]
#21 63.59 /usr/share/dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(1103,5): error MSB4018:    at System.Linq.ThrowHelper.ThrowNoElementsException() [/src/EDAPI/SyzunaPrograms.ED.EDAPI/SyzunaPrograms.ED.EDAPI.csproj]
#21 63.59 /usr/share/dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(1103,5): error MSB4018:    at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source) [/src/EDAPI/SyzunaPrograms.ED.EDAPI/SyzunaPrograms.ED.EDAPI.csproj]
#21 63.59 /usr/share/dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(1103,5): error MSB4018:    at Microsoft.NET.Build.Tasks.ValidateExecutableReferences.ExecuteCore() [/src/EDAPI/SyzunaPrograms.ED.EDAPI/SyzunaPrograms.ED.EDAPI.csproj]
#21 63.59 /usr/share/dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(1103,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/src/EDAPI/SyzunaPrograms.ED.EDAPI/SyzunaPrograms.ED.EDAPI.csproj]
#21 63.59 /usr/share/dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(1103,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/src/EDAPI/SyzunaPrograms.ED.EDAPI/SyzunaPrograms.ED.EDAPI.csproj]
#21 63.59 /usr/share/dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(1103,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/src/EDAPI/SyzunaPrograms.ED.EDAPI/SyzunaPrograms.ED.EDAPI.csproj]
#21 ERROR: process "/bin/sh -c dotnet publish -c Release --no-build -o /app/publish" did not complete successfully: exit code: 1

EDIT: can confirm this seems to be another issue related to QEMU and .NET 7.
I used a self hosted ARM64 runner and it builds perfectly fine but not on a github hosted runner using QEMU