HTTP2: Performance issues when many concurrent requests

[JamesNK]

HttpClient has performance issues when many concurrent calls are made on one connection. HttpClient is slower compared to Grpc.Core (a gRPC client that uses chttp2 native C library for HTTP/2).

Test app: https://github.com/JamesNK/Http2Perf

I have replicated a gRPC call being made using HttpClient to avoid complication from involving Grpc.Net.Client.

.NET version:

.NET SDK (reflecting any global.json):
 Version:   5.0.100-preview.4.20216.32
 Commit:    c492fe1552

HttpClient results:

• 100 concurrent callers on one HTTP/2 connection with HttpClient (dotnet run -c Release -p GrpcSampleClient r 100 false): 19k RPS, 1-6ms latency
• 100 concurrent callers on 100 HTTP/2 connections with HttpClient (dotnet run -c Release -p GrpcSampleClient r 100 true): 31k RPS, 1-8ms latency

Grpc.Core results:

• 100 concurrent callers on one HTTP/2 connection with Grpc.Core (dotnet run -c Release -p GrpcSampleClient c 100 false): 59k RPS, 1-2ms latency
• 100 concurrent callers on 100 HTTP/2 connections with Grpc.Core (dotnet run -c Release -p GrpcSampleClient c 100 true): 45k RPS, 1-2ms latency

Interesting that a connection per caller increases performance with HttpClient but decreases it with Grpc.Core.

With one caller HttpClient is faster than Grpc.Core (about 3k RPS vs 3.5k RPS) but as you can see in a 100 concurrent caller scenario Grpc.Core is three times faster.

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[scalablecory]

Are you benchmarking against Kestrel?

I suspect one large contributor to this is how we process frames sequentially: it does not max out multi-threaded CPUs very well. I've been thinking about a better buffering strategy to enable parallelism, but haven't had time to prototype it yet.

I also think our HPACK decoder can be optimized. It takes up the bulk of the CPU usage for the basic protocol parsing.

[JamesNK]

Yes it is against Kestrel. Out of curiosity I've just added a Grpc.Core server to see what changes.

The source code is here if you're curious about any of the implementation: https://github.com/JamesNK/Http2Perf

HttpClient, 100 callers, 1 connection:
Kestrel server: 18k RPS, 5ms latency
CCore server: 17k RPS, 5ms latency

CCore client, 100 callers, 1 connection:
Kestrel server: 66k RPS, 2ms latency
CCore server: 42k RPS, 2ms latency

On the server-side Kestrel is out performing CCore server. It doesn't make much difference when HttpClient is the client, an indication that HttpClient is the throughput bottleneck. When CCore client is used you can see a 50% RPS increase when switching from CCore server to Kestrel.

[scalablecory]

To expand a little on what I said previously,

Http2Connection.SendHeadersAsync:

• Fully serialized, so we have no chance for parallelism when writing headers. We can look at finer-grained locking, but keep in mind that this will prevent HPACK dynamic table encoding.
• The way we write the headers frame isn't great, it demands buffering and then copying. We can look at Consider adding Scatter/Gather IO on Stream #25344 and Finalize Socket API upgrade #33417 to make this zero-copy, or we can update our HPACK encoder to chunk frames out for us.
  • SendStreamDataAsync very similarly copies content buffers and could be optimized the same way.

ProcessIncomingFramesAsync:

• Also fully serialized, but supporting HPACK dynamic table makes this one far more difficult to process in parallel, especially when you have some servers that just pack every single header into the dynamic table. If we were to set our maximum dynamic table size to 0 again, it would open some options.
• I think the easiest win here will come from optimizing HPackDecoder, which currently does a lot of byte-at-a-time processing, copying, and string allocations. We can also shortcut static table indexes into known headers / known values.

Our windowing algorithm is also very latency-sensitive, but will primarily exhibit issues only when downloading more data than fits into the window. I don't think this will effect perf of small requests.

[karelz]

@JamesNK how close does Stephen's PR get us to reasonable perf? What is the priority of pushing perf further? Are there any customers / benchmarks demonstrating we have significant deficiencies?

[stankovski]

@karelz we have an Azure service that is affected by this perf degradation. I'll be happy to chat offline to describe our scenario as well as try to validate the fix.

[JamesNK]

@JamesNK how close does Stephen's PR get us to reasonable perf? What is the priority of pushing perf further?

I haven't had a chance to measure it yet. I am improving the gRPC benchmarks to capture more client data, and adding a golang gRPC client to compare against. That will give us two points of reference. I aim to provide new numbers next week.

When I've seen Stephen's improvements flow through to nightly builds I'll ask the customers who have raised client per issues (one is @stankovski) to retest. Right now they are considering going with Grpc.Core solely because of the perf difference.

Priority wise, I'd like the .NET gRPC client to be competitive with other gRPC client libraries. Client perf is important in microservice scenarios because it is common to have one caller rather than one thousand, and client perf can be the RPS bottle neck. We don't have to be the fastest client, but I don't want to be the slowest. Having a second point of reference will give us more information on where we stand.

[karelz]

Great, let's see where we stand with the latest fix and then we can decide how much more we need to invest in .NET 5.