CPU usage slowly increases over time

[SteveDesmond-ca]

I had previously used Application Insights, but no longer do, however the App Insights code remained in my app. I noticed when I recently migrated hosts and upgraded to .NET Core 2.1.0-rc1 that the CPU usage gradually increased over the course of a few days.

The drops back to 0 are Docker container restarts.

Removing Application Insights from the site results in a steady almost-zero CPU usage.

See also this Twitter thread, though there's not much more there.

Repro Steps

1. Add Application Insights (2.3.0-beta2) to an ASP.NET Core MVC app via NuGet and Startup.cs
   app.AddApplicationInsightsTelemetry("{no-longer-valid instrumentation key}")
2. Setup some sort of synthetic monitoring to hit the site every minute
3. Wait a couple days ¯_(ツ)_/¯

Actual Behavior

• CPU slowly creeps up over the course of a few days

Expected Behavior

• CPU remains stable

Version Info

SDK Version : 2.1.0-rc1
.NET Version : 2.1.0-rc1
How Application was onboarded with SDK(VisualStudio/StatusMonitor/Azure Extension) :
OS : Ubuntu 16.04
Hosting Info (IIS/Azure WebApps/ etc) : Azure VM

[cijothomas]

Thanks for reporting. We'll try to setup a repro and investigate more.
What was the version of application insights SDK which you used?

[SteveDesmond-ca]

2.3.0-beta2, running on an ASP.NET Core 2.1-rc1 app.

…

On Wed, May 23, 2018, 3:08 PM Cijo Thomas ***@***.***> wrote: Thanks for reporting. We'll try to setup a repro and investigate more. What was the version of application insights SDK which you used? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#690 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL9OE-fJkMOt_09ld_JL3Tl9Vhj0MMoks5t1bPEgaJpZM4UFhGL> .

[cijothomas]

Left an app in .net 2.0 running for last 3 days. Dont see any increase in mem/cpu

[cijothomas]

Similar for app in .net 2.1 for last 3 days.

[cijothomas]

I'll setup something for linux now.

[thohng]

We got same issue on Ubuntu 16.04

• After upgrade AspNetCore 2.1.0-rc1
• Never got CPU issue with AspNetCore 2.0.x before
• Not sure this caused by new AspNetCore 2.1.0-rc1 or new ApplicationInsights 2.3.0-beta2
• Internally, we got few HttpClient failed a day, might related to https://github.com/dotnet/corefx/issues/29285

Before Upgrade:

<PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.2.1" />
<PackageReference Include="Microsoft.AspNetCore.ApplicationInsights.HostingStartup" Version="2.0.2" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.0.3" />

After:

<PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.3.0-beta2" />
<PackageReference Include="Microsoft.AspNetCore.ApplicationInsights.HostingStartup" Version="2.1.0-rc1-final" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.1.0-rc1-final" />

[cijothomas]

Thanks for reporting this. We will setup a linux test to do deeper investigation soon.

[cijothomas]

Have an asp.net core 2.1 app with AI SDK 2.3, running in ubuntu 16_04. A script is pinging the app once every 10 sec.
WIll let this run for few hours/day and check again.

[cijothomas]

6hours, no visible change in CPU load. CPU is constant around .39%
Load is 6Request/Sec
Will leave this overnight.

[thohng]

When we stop pinging, the CPU usage is no more keep constantly increase.

[mehmetilker]

Same thing happening on Windows as well (Azure App service)
Not only CPU but also Garbage collector affected.
I upgrade my .Net core 2.0 site to .Net core 2.1 on Jun 3 afternoon. And yesterday I have removed AI Insight extension from my app service (Nuget package reference still in my project.)
You can see changes in attached images.

[cijothomas]

repro-ed on linux vm. Need to investigate more.

[flagbug]

@SteveDesmond-ca @mehmetilker @thohng I've seen a similar thing since upgrading to .NET Core 2.1 and not only does our CPU usage increase over time, but our open handles increase dramatically since the upgrade:

Could you take a look at your handle count and confirm if you have the same issue or not?

[cwe1ss]

I'm recently having issues with increased memory usage. I checked it with dotMemory and I've seen many System.Net.Http related objects - e.g. open handlers that haven't been closed etc.

I'm not yet sure if it's Application Insights though (I will do some more tests next week) - but if it is, maybe this clue helps finding the issue.