Failing Test: RequestTimesOutWhenRequestBodyNotReceivedAtSpecifiedMinimumRate

[Tratcher]

This is the flakiest kestrel test on the CI right now.
http://aspnetci/project.html?projectId=XPlat&buildTypeId=&tab=testDetails&testNameId=3964293374486126820&order=TEST_STATUS_DESC&branch_XPlat=__all_branches__&itemsCount=50

It started on April 14th and has 38 failures. It does not appear to be OS specific except that there are no failures on Mac.

System.TimeoutException : The operation at /_/test/shared/TestConnection.cs:129 timed out after reaching the limit of 60000ms.
   at Microsoft.AspNetCore.Testing.TaskExtensions.<TimeoutAfter>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Testing.TestConnection.<Receive>d__21.MoveNext() in /_/test/shared/TestConnection.cs:line 131
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.RequestBodyTimeoutTests.<RequestTimesOutWhenRequestBodyNotReceivedAtSpecifiedMinimumRate>d__0.MoveNext() in /_/test/Kestrel.FunctionalTests/RequestBodyTimeoutTests.cs:line 59
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[benaadams]

It does not appear to be OS specific.

Isn't a 2.1 stack trace; is it TargetFramework specific?

[Tratcher]

No, I've seen both versions of the stack:

System.TimeoutException : The operation at /_/test/shared/TestConnection.cs:129 timed out after reaching the limit of 60000ms.
   at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.Testing.TestConnection.Receive(String[] lines) in /_/test/shared/TestConnection.cs:line 131
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.RequestBodyTimeoutTests.RequestTimesOutWhenRequestBodyNotReceivedAtSpecifiedMinimumRate() in /_/test/Kestrel.FunctionalTests/RequestBodyTimeoutTests.cs:line 59
--- End of stack trace from previous location where exception was thrown ---

http://aspnetci/viewLog.html?buildId=459453&tab=buildResultsDiv&buildTypeId=Lite_KestrelStressTest#testNameId3964293374486126820

[Tratcher]

@pakrym are you still looking at this?

[pakrym]

No

[Tratcher]

Disregard the failure from 20 Mar, that was a general issue that has been fixed.

Alternate error message from:
http://aspnetci/viewLog.html?buildId=454277&tab=buildResultsDiv&buildTypeId=Lite_KestrelStressTest#testNameId3964293374486126820

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 127.0.0.1:62927
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at Microsoft.AspNetCore.Testing.TestConnection.CreateConnectedLoopbackSocket(Int32 port, AddressFamily addressFamily) in /_/test/shared/TestConnection.cs:line 260
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.TestServer.CreateConnection() in /_/test/Kestrel.FunctionalTests/TestHelpers/TestServer.cs:line 115
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.RequestBodyTimeoutTests.<RequestTimesOutWhenRequestBodyNotReceivedAtSpecifiedMinimumRate>d__0.MoveNext() in /_/test/Kestrel.FunctionalTests/RequestBodyTimeoutTests.cs:line 35
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

All other failures have the same timeout error from the original post above.

[Tratcher]

39 Additional failures in Release/2.1 from 24-Jan to current.
http://aspnetci/project.html?projectId=Releases_21Public&buildTypeId=&tab=testDetails&testNameId=3964293374486126820&order=TEST_STATUS_DESC&branch_Releases_21Public=__all_branches__&itemsCount=50

There are many failures from 24-29 Jan across all OS's. These mostly look like issues with the new Memory type.
02-Feb and 29 Mar are both issues with the logging infrastructure.

Recent failures since 17-Apr have been on Ubuntu and all 6 match the original timeout error above. These are all netcoreapp2.0 failures.

[mikeharder]

Also failed on my dev machine:

Starting test group: Libuv.FunctionalTests/netcoreapp2.1

System.TimeoutException : The operation at D:\Git\KestrelHttpServer\test\shared\TestConnection.cs:129 timed out after reaching the limit of 60000ms.
Stack Trace:
  at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber)
  at Microsoft.AspNetCore.Testing.TestConnection.Receive(String[] lines) in D:\Git\KestrelHttpServer\test\shared\TestConnection.cs:line 131
  at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.RequestBodyTimeoutTests.RequestTimesOutWhenRequestBodyNotReceivedAtSpecifiedMinimumRate()
in D:\Git\KestrelHttpServer\test\Kestrel.FunctionalTests\RequestBodyTimeoutTests.cs:line 56

[mikeharder]

Another repro: https://travis-ci.org/aspnet/KestrelHttpServer/jobs/380397601

[mikeharder]

All recent failures:

http://aspnetci/project.html?tab=testDetails&testNameId=3964293374486126820&projectId=AspNetEf&order=TEST_STATUS_DESC

Most common stack:

System.TimeoutException : The operation at /_/test/shared/TestConnection.cs:131 timed out after reaching the limit of 60000ms.
   at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber)
   at Microsoft.AspNetCore.Testing.TestConnection.Receive(String[] lines) in /_/test/shared/TestConnection.cs:line 133
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.RequestBodyTimeoutTests.RequestTimesOutWhenRequestBodyNotReceivedAtSpecifiedMinimumRate() in /_/test/Kestrel.FunctionalTests/RequestBodyTimeoutTests.cs:line 59
--- End of stack trace from previous location where exception was thrown ---

It seems the test fails most often (or only) in test group Libuv.FunctionalTests/netcoreapp2.2.

Unable to repro when running just this test in a loop. Passes 100/100.

cmd /c 'for /l %n in () do dotnet test --no-build -f netcoreapp2.2 --filter "DisplayName~RequestTimesOutWhenRequestBodyNotReceivedAtSpecifiedMinimumRate" 2>&1'

Environments tested:

• Dev machine
• Hyper-V VM, all CPUs, Win10
• Hyper-V VM, 1 CPU, Win10
• Hyper-V VM, all CPUs, Ubuntu 18.04