-
Notifications
You must be signed in to change notification settings - Fork 5.1k
Description
Description
I'm encountering random segmentation faults (and sometimes AccessViolationException and NullReferenceException) when running a dotnet 8.0 console app on 32-bit ARM linux. It seems to mainly occur for me when it attempts to connect to a SignalR hub as a client.
Reproduction Steps
I have pushed a small repro app pair here.
Note that in addition to the crashing console app, it also contains a trivially simple SignalR webapp that I've deployed to azure when reproducing. (Note the "[CHANGE THIS TO YOUR WEBAPP URL]" line in the console app)
The console app code just tries to make a SignalR connection over websockets and send/receive a few MessagePack messages before exiting.
When running the console app repeatedly, it has a random chance of encountering the issue, after something like 10 to 100 attempts. (a helper bash loop .sh is included)
Expected behavior
Can run repeatedly without crashing.
Actual behavior
Approximately 5% of the time it fails with simply Segmentation fault
.
I have uploaded a core crash dump file of one of the segfault occurrences here
On some occasions it randomly has a NullReferenceException, with additional stack trace info:
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.GetStateMachineBox[TStateMachine](TStateMachine& stateMachine, Task`1& taskField)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.HandshakeAsync(ConnectionState startingConnectionState, Int32 protocolVersion, CancellationToken cancellationToken)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncCore(CancellationToken cancellationToken)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncCore(CancellationToken cancellationToken)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncInner(CancellationToken cancellationToken)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsync(CancellationToken cancellationToken)
at Program.<Main>$(String[] args)
at Program.<Main>(String[] args)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AwaitUnsafeOnCompleted[TAwaiter](TAwaiter& awaiter, IAsyncStateMachineBox box)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
Fatal error. Internal CLR error. (0x80131506)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Threading.Tasks.Task+<>c.<ThrowAsync>b__128_1(System.Object)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
Regression?
It seems to work fine (even with thousands of attempts) when changing the csproj back to dotnet 6.0 instead of 8.0.
(I can't test dotnet 7.0 on this ARM device due to the higher glibc requirement 7.0 has)
Known Workarounds
Downgrading the csproj to dotnet 6.0
Configuration
Custom linux OS running on 32-bit ARM linux IoT device.
Problem seems specific to 32-bit ARM. Ran repeatedly on a different custom ARM64 device with no issues.
Other information
On some rare occasions with very similar code (from my team's proprietary app), it encountered this AccessViolationException, which I suspect is stemming from the same cause:
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ClearStateUponCompletion()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
at System.Threading.Tasks.Task.RunContinuations(System.Object)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<Int32>, Int32)
at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Int32)
at System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
at System.Threading.ThreadPool+<>c.<.cctor>b__52_0(System.Object)
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SignalCompletion()
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Int32)
at System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
at System.Threading.ThreadPool+<>c.<.cctor>b__52_0(System.Object)
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1[[System.Boolean, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SignalCompletion()
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1[[System.Boolean, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Boolean)
at System.Net.Sockets.SocketAsyncEventArgs.OnCompletedInternal()
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32, System.Net.Sockets.SocketFlags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError)
at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, System.Memory`1<Byte>, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError)
at System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ProcessAsyncOperation(System.__Canon)
at System.Net.Sockets.SocketAsyncContext+ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents)
at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()