Remove spin lock from SocketAsyncEventArgs on Windows (#64770)

* Remove spin lock from SocketAsyncEventArgs on Windows

The Windows implementation of SocketAsyncEventArgs has a spin lock to help coordinate between the thread initiating the Winsock operation and the eventual overlapped completion callback.  There are some operations we delay (e.g. registering for cancellation) until after the operation has already been initiated and shown to complete asynchronously rather than synchronously, and as these are being set up after the Winsock call to perform the socket operation, it's possible for the overlapped completion to happen before or while we do that additional cleanup.  This condition was expected to be rare, and a spin lock is used to ensure that if the race condition does occur, the callback waits for the state set up to be complete before continuing.

However, it turns out for certain operations it's actually not that rare.  In particular, it appears that accepts when there's already a pending connection end up frequently completing asynchronously but immediately, which causes this race condition to manifest, and we've seen the spin lock spin so many times that it falls back to an actual sleep that causes unexpected delays.

We can instead just maintain a simple gate that describes whether the launching thread or the callback thread own completion.  The launcher sets up all the state and then tries to transition to set the gate.  Similarly, the first thing the callback does is set the gate (to a packed result in case the launcher needs it).  Whoever gets there second is responsible for handling completion.  If the launching thread is the one that gets there second, it essentially turns the asynchronous operation into a synchronous one, from the perspective of the caller, just as if the operation had completed synchronously.

* Avoid an Interlocked.Exchange in the common case

Author: stephentoub