Description
After Kestrel update from 7.0.0-rc.1.22376.12 to 7.0.0-rc.1.22377.5, HttpClient benchmarks started showing ~2.5% failures (there was 0 exceptions before). Elevated number of failures can be seen on stress tests as well; the last successful (0 exceptions) run was on 7.0.0-rc.1.22368.6 link to Run #20220727.4, the following run with ~14% failures is on 7.0.0-rc.1.22377.5 link to Run #20220728.2.
Errors on benchmark are
[05:58:25.603] Exception: System.Net.Http.HttpRequestException: The server returned an invalid or unrecognized response.
at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at HttpClientBenchmarks.Program.Measure(Func`1 sendAsync) in C:\Users\Administrator\AppData\Local\Temp\benchmarks-agent\benchmarks-server-6180\t4ofjfuz.2si\benchmarks\src\BenchmarksApps\HttpClientBenchmarks\Clients\HttpClient\Program.cs:line 359
This might not be related to #72739, because this happens on successful scenarios (no Abort, no cancellations) and fix from Mana's PR #72746 doesn't help (whereas it did help with #72739).
NOTE: Below, runtime version is fixed and the same for all runs for both client and server. Changed is only server side ASP.NET Core version.
HttpClient benchmarks commands
Before
crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --client.framework net7.0 --client.channel latest --server.framework net7.0 --server.channel latest --profile aspnet-perf-win --scenario httpclient-kestrel-get --variable useHttps=true --variable httpVersion="3.0" --variable responseSize=8192 --variable concurrencyPerHttpClient=1000 --client.sdkVersion 7.0.100-rc.1.22381.2 --client.aspNetCoreVersion 7.0.0-rc.1.22401.1 --client.runtimeVersion 7.0.0-rc.1.22401.1 --server.sdkVersion 7.0.100-rc.1.22377.4 --server.aspNetCoreVersion 7.0.0-rc.1.22376.12 --server.runtimeVersion 7.0.0-rc.1.22401.1
| Exceptions | 0 |
| Mean RPS | 24,213 |
After
crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --client.framework net7.0 --client.channel latest --server.framework net7.0 --server.channel latest --profile aspnet-perf-win --scenario httpclient-kestrel-get --variable useHttps=true --variable httpVersion="3.0" --variable responseSize=8192 --variable concurrencyPerHttpClient=1000 --client.sdkVersion 7.0.100-rc.1.22381.2 --client.aspNetCoreVersion 7.0.0-rc.1.22401.1 --client.runtimeVersion 7.0.0-rc.1.22401.1 --server.sdkVersion 7.0.100-rc.1.22381.2 --server.aspNetCoreVersion 7.0.0-rc.1.22377.5 --server.runtimeVersion 7.0.0-rc.1.22401.1
| Exceptions | 494 |
| Mean RPS | 21,634 |
The commit comparison for Kestrel between 7.0.0-rc.1.22376.12 and 7.0.0-rc.1.22377.5: dotnet/aspnetcore@6df6573...64744fd
cc @JamesNK
P.S.:
Additionally, after the same update, Kestrel seemed to strangely react to setting max number of HTTP/3 streams, seems like whatever number is put there, the RPS becomes extremely limited (by 99,9%, ~20K->14)
HttpClient benchmarks commands
Before, with http3StreamLimit=65535
crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --client.framework net7.0 --client.channel latest --server.framework net7.0 --server.channel latest --profile aspnet-perf-win --scenario httpclient-kestrel-get --variable useHttps=true --variable httpVersion="3.0" --variable responseSize=8192 --variable concurrencyPerHttpClient=1000 --client.sdkVersion 7.0.100-rc.1.22381.2 --client.aspNetCoreVersion 7.0.0-rc.1.22401.1 --client.runtimeVersion 7.0.0-rc.1.22401.1 --server.sdkVersion 7.0.100-rc.1.22377.4 --server.aspNetCoreVersion 7.0.0-rc.1.22376.12 --server.runtimeVersion 7.0.0-rc.1.22401.1
| Exceptions | 0 |
| Mean RPS | 18,887 |
After, with http3StreamLimit=65535
crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --client.framework net7.0 --client.channel latest --server.framework net7.0 --server.channel latest --profile aspnet-perf-win --scenario httpclient-kestrel-get --variable useHttps=true --variable httpVersion="3.0" --variable responseSize=8192 --variable concurrencyPerHttpClient=1000 --client.sdkVersion 7.0.100-rc.1.22381.2 --client.aspNetCoreVersion 7.0.0-rc.1.22401.1 --client.runtimeVersion 7.0.0-rc.1.22401.1 --server.sdkVersion 7.0.100-rc.1.22381.2 --server.aspNetCoreVersion 7.0.0-rc.1.22377.5 --server.runtimeVersion 7.0.0-rc.1.22401.1
| Exceptions | 45 |
| Mean RPS | 14 |
The code how max number of streams is set in benchmarks: https://github.com/aspnet/Benchmarks/blob/1c1078e89f6c66b6cd1d4463f3120bb89d4b4ac5/src/BenchmarksApps/HttpClientBenchmarks/Servers/Kestrel/Program.cs#L89-L95