Kestrel fails to respond when used with HAProxy

[rogeralsing]

I am trying to use Asp.NET Core with Rancher Lab's docker orchestration.
If I just deploy my dockerised asp.net core container in my rancher cluster, everything works fine.
The service responds as expected.

But, if I add a Rancher load balancer (HA Proxy) infront of it.
I get the following errors when processing requests.
The request eventually goes through, but takes several seconds and there are multiple errors appearing in the logs before it goes through, so my guess is that HaProxy retries and retries untill it succeeds... but the question then is why does it fail in the first place?

Placing the same predefined balancer infront of any other web service works fine.
e.g. any of my Go lang services.
So my guess is that the problem in Kestrel

06/04/2016 22:53:25�[97m      Connection processing ended abnormally
06/04/2016 22:53:25      System.IO.IOException: Error -104 ECONNRESET connection reset by peer ---> Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -104 ECONNRESET connection reset by peer
06/04/2016 22:53:25        --- End of inner exception stack trace ---
06/04/2016 22:53:25        at Microsoft.AspNet.Server.Kestrel.Http.SocketInput.GetResult () [0x00000] in <filename unknown>:0
06/04/2016 22:53:25        at Microsoft.AspNet.Server.Kestrel.Http.SocketInputExtensions+<ReadAsync>d__0.MoveNext () [0x00000] in <filename unknown>:0
06/04/2016 22:53:25      --- End of stack trace from previous location where exception was thrown ---
06/04/2016 22:53:25        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0
06/04/2016 22:53:25        at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x00000] in <filename unknown>:0
06/04/2016 22:53:25        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00000] in <filename unknown>:0
06/04/2016 22:53:25        at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00000] in <filename unknown>:0
06/04/2016 22:53:25        at System.Runtime.CompilerServices.TaskAwaiter`1[System.Int32].GetResult () [0x00000] in <filename unknown>:0
06/04/2016 22:53:25        at Microsoft.AspNet.Server.Kestrel.Http.Frame+<RequestProcessingAsync>d__79.MoveNext () [0x00000] in <filename unknown>:0 �[39m
06/04/2016 22:53:25�[33mwarn�[39m�[37m: Microsoft.AspNet.Server.Kestrel[0]�[39m
06/04/2016 22:53:25�[97m      Connection shutdown abnormally
06/04/2016 22:53:25      System.IO.IOException: Error -104 ECONNRESET connection reset by peer ---> Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -104 ECONNRESET connection reset by peer
06/04/2016 22:53:25        --- End of inner exception stack trace ---
06/04/2016 22:53:25        at Microsoft.AspNet.Server.Kestrel.Http.SocketInput.GetResult () [0x00000] in <filename unknown>:0
06/04/2016 22:53:25        at Microsoft.AspNet.Server.Kestrel.Http.Frame+<RequestProcessingAsync>d__79.MoveNext () [0x00000] in <filename unknown>:0 �[39m
06/04/2016 23:00:39�[32minfo�[39m�[37m: Microsoft.AspNet.Hosting.Internal.HostingEngine[1]�[39m
06/04/2016 23:00:39�[97m      Request starting HTTP/1.1 GET http://13.94.232.63:8080/api/health  �[39m
06/04/2016 23:00:39�[32minfo�[39m�[37m: Microsoft.AspNet.Mvc.Controllers.ControllerActionInvoker[1]�[39m
06/04/2016 23:00:39�[97m      Executing action method AspNetCoreService.Controllers.HealthController.Get with arguments () - ModelState is Valid'�[39m
06/04/2016 23:00:39�[32minfo�[39m�[37m: Microsoft.AspNet.Mvc.Infrastructure.ObjectResultExecutor[1]�[39m
06/04/2016 23:00:39�[97m      Executing ObjectResult, writing value Microsoft.AspNet.Mvc.ActionContext.�[39m
06/04/2016 23:00:39�[32minfo�[39m�[37m: Microsoft.AspNet.Mvc.Infrastructure.MvcRouteHandler[2]�[39m
06/04/2016 23:00:39�[97m      Executed action AspNetCoreService.Controllers.HealthController.Get in 0.0001ms�[39m
06/04/2016 23:00:39�[32minfo�[39m�[37m: Microsoft.AspNet.Hosting.Internal.HostingEngine[2]�[39m
06/04/2016 23:00:39�[97m      Request finished in 0.0001ms 200 text/plain; charset=utf-8�[39m

[davidfowl]

Are you using mono?

[rogeralsing]

Im using the official microsoft/aspnet docker image as my base, so no (unless there is some dnx mono magic that I havent understood yet?)

FROM microsoft/aspnet
ADD . /app
WORKDIR /app
RUN dnu restore
EXPOSE 5000/tcp
ENTRYPOINT ["dnx", "web", "--server.urls", "http://0.0.0.0:5000"]

(The image exists at https://hub.docker.com/r/rogeralsing/aspnetcoredemo/
And there is a GET endpoint at /api/health if anyone wants to try)

[rogeralsing]

I found similar questions regarding HAProxy and Netty.
http://stackoverflow.com/questions/21550337/haproxy-netty-way-to-prevent-exceptions-on-connection-reset

Can this be the same problem they saw in Netty?

"Now the problem is that when haproxy closes a connection for means of rebalancing, it does so by sending a tcp-RST."

[paralin]

Same thing in NGINX, it's already fixed, but there's been no release containing the fix.

#341

[halter73]

@paralin I think your right that this is the same issue.

@rogeralsing Can you try enabling HAProxy's http-keep-alive option or using the nightly packages from the https://www.myget.org/F/aspnetcidev/api/v3/index.json NuGet feed? Doing either should fix the issue.

Note: The Microsoft.AspNet.* packages have been renamed to Microsoft.AspNetCore.*.

[rogeralsing]

Thanks, are there any pointers to how to set VS.NET up to use the new nuget feed?
just adding that to the nuget sources seems to break things, System.Threading.Task found in multiple libs etc.

[spboyer]

@rogeralsing I used the haproxy image that tutum (no docker) created when I hit the same issue. It is a haproxy cfg setting / config somewhere that is beyond what I could find documented. But this docker image fixed it for me. https://github.com/tutumcloud/haproxy

Here is my post: http://www.tattoocoder.com/legion-of-heroes-haproxy-nginx-angular2-aspnetcore-redis-docker/

[halter73]

@rogeralsing Running the rc2 with dnx might not work atm. @davidfowl might know better.

If you want to use rc2 bits, you probably are going to need to use the same version of the dotnet cli we're using to build rc2. Currently that's 1.0.0-beta-002279, but that can change daily.

It might be easier to continue using rc1 for now and enabling the http-keepalive option if that works.

[cesarblum]

@rogeralsing You can find the current CLI versions we're using for each OS in the cli.* files here: https://github.com/aspnet/KoreBuild/tree/dev/build

[amcdnl]

I'm still seeing this in RC2

[mikeharder]

@amcdnl: I believe the problems you were seeing were resolved in #636, correct?

[amcdnl]

@mikeharder Yes :)