AspNet Core 1.1 on Azure - web app works for a while, then later I get "CGI application encountered an error"

[dotnetdummy]

This is basically a copy of issue #576 and #602 since I have still see this error today on an Azure hosted AppService and Asp.Net Core 1.1.

I've tried with suggested solutions like removing "forwardWindowsAuthToken" (found here) and by creating a new AppService aswell.

What I've noticed is that the problem seem to occurs after running load tests on the AppService (and sometimes after a deploy aswell).

The WebApp is configured as "AlwaysOn" on a Standard S1 with 4 instances.

[muratg]

@shirhatti Could you take a look?

[shirhatti]

@Cbird Can you share your event logs and Kestrel logs?

[dotnetdummy]

@shirhatti we are doing some load tests the coming days on the application. I will gather some logs for you then.

[dotnetdummy]

@shirhatti I did some more load tests with 500 concurrent users during 5min. The operation behind the url requested is basically a middleware forwarding the request to another application (through the use of HttpClient with 1000 MaxConnectionsPerServer). The other application is made in .Net 4.5.1 WebAPI2 and returns a static json payload.

When running the loadtests on the Asp.Net Core 1.1 application it fails 30% of the 3578 requests it manages to fire off (with avg responsetime of 35s). When running directly on the other .Net 4.5.1 application it succeeds with 100%, fires off 137075 requests with avg responsetime of 0.06s.

During these tests I tried accessing the same endpoint and the error 502 "The specified CGI application encountered an error and the server terminated the process." occurred again. This request took 121532ms and below is the event logs I got from the Kudo "Azure App Service Support":

Event: Error
Event ID: 2284
Source: W3SVC-WP
Details:
1
5
50000780

I feel the details is quite vague (but perhaps it's helpful for you). I also activated Diagnostic Logs -> Application Logging but nothing was dumped (and it deactivates itself after each load test). Application Insights is also activated, but it reports all is fine (simply because the failing requests doesn't even reach the application).

If there is anything else you need just ping me.

[dotnetdummy]

Further information. When scaling down the application from 4 instances to 2 the load tests went through without any errors.

[shirhatti]

/cc @pan-wang

[muratg]

@pan-wang @shirhatti thoughts? Would getting a dump help?

[shirhatti]

This request took 121532ms

My suspicion looking at that number (~2 minutes) which the default request timeout in ANCM is that there is probably a deadlock in your backend.
@Cbird Is there any chance you can share a dump file when you next repro this?

[dotnetdummy]

Hey @shirhatti. I've converted the application back to Asp.Net 4.6.1 in order to proceed with the project.

Now it works as expected, and the biggest differences (expect from the obvious framework change) is the caching and DI container. In the Asp.Net Core app I used the DI and caching implementations in Asp.Net Core, and in the Asp.Net 4.6.1 app I used an abstraction of the MemoryCache object and DryIoc as dependency injector.

It being a deadlock seems logical since, as you said, the timeout in ANCM correlates with the number I see. And then this perhaps is related to either the cache, or the DI since they are different.

I will try to reproduce it again with the other codebase again, but I don't know when I will have time to do this now.

[glennc]

Putting to discussion until this is either moved to ANCM or something actionable for hosting falls out.

[Flood]

I am also experiencing this error from time to time. It happens randomly without load tests, ~1 time/month and is solved by restarting the server in Azure portal.

I have the forwardWindowsAuthToken in my generated web.config (.net core) and I don't want to remove it since it's generated when building/compiling the application.

[davidhagg]

We are also experiencing this issue intermittently. Nothing in the event logs that could relate to this.
It could be that ANCM or the IIS is stopping the requests early before reaching Kestrel.
According to the MS support there is a known bug? Could someone from MS please comment on this?

[Sebastian-Negoescu]

Hello,

“The specified CGI application encountered an error and the server terminated the process” w/ HTTP 502.3

While the issue appears to be fixed right now in Windows Server, when it comes to Azure App Services, we can discuss about the following:

• The ANCM fix is being deployed as part of Antares v.64;
• Ant64 is currently rolling out and I got a confirmation that it should be completed by Monday.

[aspnet-hello]

We periodically close 'discussion' issues that have not been updated in a long period of time.

We apologize if this causes any inconvenience. We ask that if you are still encountering an issue, please log a new issue with updated information and we will investigate.