SocketsHttpHandler may read request Content multiple times in parallel

[davidni]

Description

Under very specific conditions, SocketsHttpHandler.SendAsync when using HTTP/2 attempts to copy the HttpRequestMessage.Content twice.

In Island Gateway (AKA what eventually became YARP), we have code similar to the following, and this invariant that we do a best-effort check is being violated:

https://github.com/microsoft/reverse-proxy/blob/961608d01adc9ab07d4cd3f5adf7d4d3dfff3f0f/src/ReverseProxy/Proxy/StreamCopyHttpContent.cs#L118-L125

The following criteria seem to be necessary to trigger this bug, though it may be broader than this:

• Request specifies Expect: 100-continue
• Destination does not produce a 100 response within the first 1 second
• HttpConnectionPool.SendWithRetryAsync decides to retry (just a hypothesis)

It seems like the SocketsHttpHandler machinery may be deciding it is safe to retry because the destination service provably did not start processing the first attempt, but it seems to make the wrong assumption that it is okay to read the HttpRequestMessage.Content again on the next attempt. This is not a valid assumption, and leads to issues in applications such as YARP where the Content is a wrapper around Kestrel's incoming request body stream, which cannot be read twice.

Configuration

• .NET Core 3.1.15
• Windows Server Datacenter 2019

Other information

Filing this as a result of an offline discussion with @halter73 and @davidfowl. Thanks folks for helping investigate.

@karelz FYI.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Under very specific conditions, SocketsHttpHandler.SendAsync when using HTTP/2 attempts to copy the HttpRequestMessage.Content twice.

In Island Gateway (AKA what eventually became YARP), we have code similar to the following, and this invariant that we do a best-effort check is being violated:

https://github.com/microsoft/reverse-proxy/blob/961608d01adc9ab07d4cd3f5adf7d4d3dfff3f0f/src/ReverseProxy/Proxy/StreamCopyHttpContent.cs#L118-L125

The following criteria seem to be necessary to trigger this bug, though it may be broader than this:

• Request specifies Expect: 100-continue
• Destination does not produce a 100 response within the first 1 second
• HttpConnectionPool.SendWithRetryAsync decides to retry (just a hypothesis)

It seems like the SocketsHttpHandler machinery may be deciding it is safe to retry because the destination service provably did not start processing the first attempt, but it seems to make the wrong assumption that it is okay to read the HttpRequestMessage.Content again on the next attempt. This is not a valid assumption, and leads to issues in applications such as YARP where the Content is a wrapper around Kestrel's incoming request body stream, which cannot be read twice.

Configuration

• .NET Core 3.1.15
• Windows Server Datacenter 2019

Other information

Filing this as a result of an offline discussion with @halter73 and @davidfowl. Thanks folks for helping investigate.

@karelz FYI.

Author:	davidni
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[geoffkizer]

Can you try with a recent .NET 6 build?

There are a bunch of situations that can cause us to try to read the content stream twice. If the content doesn't support this, it can throw and we should fail the request. There's not really anything else we can do at that point.

[geoffkizer]

More info:

Things that can cause the content stream to be read twice include stuff like redirects and authentication. We will handle specific status codes like 302 or 401 and issue another request and try to send the content again. Expect: 100-continue can be used to try to avoid having to read the content twice in cases like this, because hopefully we will get the 302 or 401 before proceeding to send content; but obviously this only will work if the expect timeout doesn't expire.

Are you using redirects or authentication here?

We do have logic to retry requests on connection shutdown -- that's what HttpConnectionPool.SendWithRetryAsync is about. But this shouldn't cause the content stream to be read twice. Once we start sending any content, the request is no longer retryable.

[davidni]

Thanks Geoff.

1: Unfortunately I do not have a minimal repro of this atm and I am unable to verify on .NET 6 (we will not transition our production workloads before GA).

2: There are no redirects nor authentication at play here. Connection shutdown is likely the trigger -- during the incident where we had this happen, the destination service was overwhelmed and couldn't handle the load. This meant many of our requests were hitting the 1-second timeout for Expect:100-continue. And in at least a few cases, this did cause the content stream to be read twice. We have unambiguous evidence of this based on a few occurrences of the error I mentioned in the issue above ("Stream was already consumed"). We have indirect evidence of this based on Kestrel logs with the following stack trace, indicating two concurrent calls to read the request content (look for StreamCopyHttpContent.SerializeToStreamAsync below):

System.NotSupportedException:  The ReadAsync method cannot be called when another read operation is pending.
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.IO.Pipelines.StreamPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
   at IslandGateway.Core.Service.Proxy.StreamCopier.CopyAsync(Stream source, Stream destination, CancellationToken cancellation)
   at IslandGateway.Core.Service.Proxy.StreamCopyHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context)
   at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
   at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at IslandGateway.Core.Service.Proxy.HttpProxy.<>c__DisplayClass13_0.<<ExecuteRequestAsync>b__0>d.MoveNext()

[geoffkizer]

Ahhhh, this is HTTP2. I had assumed HTTP/1.1.

Yes, with HTTP2, there are additional scenarios. When we receive a GOAWAY (server-initiated connection shutdown), the server tells us which streams will be processed and which won't; the ones that haven't are retryable, even if we already started sending the content. So in this case you will definitely see us try to read the content twice.

That said, this is expected behavior. If you can't support reading the content again, then you should throw and the request should be failed.

[davidni]

@Tratcher for your take re. YARP. If confirmed this is acceptable, we would at least want to make sure we are handling it correctly and emitting the right data to telemetry. E.g. perhaps this should count as a Transport error (?)

[geoffkizer]

You can also consider increasing the Expect100ContinueTimeout so that you'll hit this less frequently.

[Tratcher]

We have indirect evidence of this based on Kestrel logs with the following stack trace, indicating two concurrent calls to read the request content

This part's concerning, it means SerializeToStreamAsync is called concurrently on two threads and the Started check in YARP isn't thread safe. The two concurrent threads make sense if the retry is happing at the same time as the 100-continue expires. @geoffkizer that sounds like a potential race in the client logic. If the 100-continue expired and is sending content then the retry should wait for that to abort.

This could be mitigated by making the Started check in YARP thread safe. I'll file an issue for that (dotnet/yarp#1068).

As for the retry scenario in general, yes that's expected to fail in YARP and it should be reported accordingly. What type of failure is it currently reported as? It might depend on if it sent the whole request body before retrying, or if it was aborted part way through.

[Tratcher]

Raising the 100-continue timeout is a valid mitigation and shouldn't affect responsive servers.

[geoffkizer]

@geoffkizer that sounds like a potential race in the client logic. If the 100-continue expired and is sending content then the retry should wait for that to abort.

Looking at the code, we should be waiting for SerializeToStreamAsync to complete before we actually fail the request. So I don't see how we could end up with concurrent SerializeToStreamAsync calls here. (I'm looking at 6.0 code, but I don't think this has changed much since 3.1.)

Note that in cases like this, where we get a GOAWAY or similar and need to cancel sending the request body, we will do two things:

(1) Cancel the cancellation token passed to SerializeToStreamAsync
(2) Throw on any attempt to use the request body stream passed to SerializeToStreamAsync