System.ArgumentException: The supplied System.Net.SocketAddress is an invalid size for the System.Net.IPEndPoint end point. (Parameter 'socketAddress') on OSX

[mdemler]

Description

We're encountering this exception periodically on OSX via two paths.

Most of the time we see this call stack:

System.ArgumentException: The supplied System.Net.SocketAddress is an invalid size for the System.Net.IPEndPoint end point. (Parameter 'socketAddress')
  ?, in EndPoint IPEndPoint.Create(SocketAddress)
  ?, in SocketError SocketAsyncEventArgs.FinishOperationAccept(SocketAddress)
  ?, in void SocketAsyncEventArgs.FinishOperationSyncSuccess(int, SocketFlags)
  ?, in void SocketAsyncEventArgs.FinishOperationAsyncSuccess(int, SocketFlags)
  ?, in void SocketAsyncEventArgs.CompletionCallback(int, SocketFlags, SocketError)
  ?, in void SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Memory<byte> socketAddress, SocketError socketError)
  ?, in void AcceptOperation.InvokeCallback(bool)
  ?, in void OperationQueue<TOperation>.ProcessAsyncOperation(TOperation)
  ?, in void SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation)
  ?, in void ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
  ?, in void AsyncOperation.Process()
  ?, in void SocketAsyncContext.HandleEvents(SocketEvents)
  ?, in void SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
  ?, in bool ThreadPoolWorkQueue.Dispatch()
  ?, in void WorkerThread.WorkerThreadStart()
  ?, in void Thread.StartCallback()

Once we also saw this call stack:

System.ArgumentException: The supplied System.Net.SocketAddress is an invalid size for the System.Net.IPEndPoint end point. (Parameter 'socketAddress')
  ?, in EndPoint IPEndPoint.Create(SocketAddress)
  ?, in SocketError SocketAsyncEventArgs.FinishOperationAccept(SocketAddress)
  ?, in void SocketAsyncEventArgs.FinishOperationSyncSuccess(int, SocketFlags)
  ?, in void SocketAsyncEventArgs.FinishOperationSync(SocketError, int, SocketFlags)
  ?, in SocketError SocketAsyncEventArgs.DoOperationAccept(Socket, SafeSocketHandle, SafeSocketHandle, CancellationToken)
  ?, in bool Socket.AcceptAsync(SocketAsyncEventArgs, CancellationToken)
  ?, in ValueTask<Socket> AwaitableSocketAsyncEventArgs.AcceptAsync(Socket, CancellationToken)
  ?, in ValueTask<Socket> Socket.AcceptAsync(Socket, CancellationToken) x 2
  ... Calls from our application

Our application implements its own approach to peer-to-peer communication. This is regularly occurring in our testing environment, where we host multiple instances of our application on the same machine, communicating with each other via different ports.

Reproduction Steps

We unfortunately haven't been able to pinpoint a regularly behavior that causes this to happen.

We appear to have only started seeing this issue once we started hosting multiple instances of our application on the same machine.

Expected behavior

While we can safely handle this exception when we're directly invoking it, the majority of the time looks to be via background processing, which we cannot. We are only aware that this exception is occurring because we have logging attached to the unhandled exception handler.

Actual behavior

This error does not occur, or it is always bubbled up to the AcceptAsync caller, such that we are able to catch and handle it.

Regression?

Not 100% sure, but don't believe that a .NET upgrade materialized this. If at all, it would certainly have been a minor one, as we've been on .NET 8 since last year.

Known Workarounds

Our application looks to successfully communicate with the peer on subsequent attempts.

Configuration

.NET 8.0.3, osx-x64

Operating System
Kernel Version: 23.1.0
Name: Darwin
Raw Description: Darwin 23.1.0 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:31 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8112

Other information

No response

[wfurt]

Do you use dual mode sockets? IPv4, IPv6, mix ...? Since you are in Accept path I assume this is TCP? Can you get core dump or get it under debugger and inspect the SocketAddress in FinishOperationAccept?

[mdemler]

Yes, this is a typical SocketType.Stream, ProtocolType.Tcp socket that we're listening on.

We're binding to an IPv4 address.

Given that this happens at random in a test environment, getting it under a debugger or obtaining a core dump is probably not going to be achievable.

[wfurt]

For the dump, I was thinking AppDomain.FirstChanceException in combination with Environment.FailFast. If the systems are not configured to collect dumps, you can also do it explicitly using https://learn.microsoft.com/en-us/dotnet/core/diagnostics/microsoft-diagnostics-netcore-client

It is going to be difficult to investigate without more insight.

[mdemler]

Understood. Might take a few weeks, but will work on getting that capability added to our app and a dump when the issue triggers over to you.

[wfurt]

one more thought: Do you run multiple instances on the same port .e.g load balancing?

I think I can inject the exception artificially and fix the path e.g. it should bubble up to the caller and not be thrown on thread pool IMHO. With that, it would at least be possible to handle it.

[wfurt]

actually, you claim it is different port, right? In that case I have no clue what another independent instance would matter.

[mdemler]

Yeah, each instance should be listening on its own port. Things do get misconfigured, but I'd expect the Bind call to fail in those instances.

In these scenarios, we've successfully bound to and are listening on the port, and we're processing connections from other peers, including this one, just fine, but we're encountering this failure for some calls to AcceptAsync.

The multiple-instance thing might be a red herring, but was the only thing I could identify that we'd changed around the time that we first started seeing this.

[liveans]

Triage: It's not actionable without a repro. We'll look into this more. Moving to the future.