.Net Core 2.0 + 2.2 High Memory Usage On Startup

[AdamRiddick]

Describe the issue

When spinning up an .Net Core Application after an application pool recycle in IIS, the process seems to go on a few memory grabbing runs but doesn't appear to be using it - This doesn't look like a memory leak as the memory is always reclaimed.

For example, the memory usage will cycle upwards to around 3-4GB, back down to about 400mb, then up and down between these a handful more times before settling down to ~200mb. The application starts as normal through this process, and can be accessed and used as normal.

Normal usage averages at around 200mb.

We've managed to take a dump at it's peak;

To Reproduce

• .Net Core 2.0
• .Net Core Runtime 2.0.7
• Windows Server 2016
• Workstation GC Mode

We haven't been able to reproduce this outside of our production server environments.

Is there a way we can determine what is causing this high memory usage during startup?

Edit
I've also been able to reproduce this with;

• .Net Core 2.2
• .Net Core Runtime 2.2.4
• Windows Server 2016
• Workstation GC Mode

[analogrelay]

We've managed to take a dump at it's peak;

Would you be willing to share the dump itself privately with us so we can examine it further?

[AdamRiddick]

Yeah we can do that. How’s best to get that to you, and do you have a preferred zip format?

[analogrelay]

You can send it to me at [my github username]@microsoft.com, though if it's large (and since it's a memory dump I'd imagine it is :)) you'll probably need to find a file sharing service (OneDrive, DropBox, Azure Blob, etc.) to host it. I should be fine with most formats, as long as 7-zip can open it ;).

[AdamRiddick]

Thanks, I've send a share link over on OneDrive :)

[AdamRiddick]

I'm not sure if its related but during general operation we are seeing some occurrences ofOutOfMemoryException being thrown on servers with multiple instances and there isn't a common stack trace among them.

[AdamRiddick]

@anurse
We've been looking at this further and we currently believe this is related to #2737.

Using the recommended workaround on that issue, to switch the ServiceProviderMode to Runtime appears to have resolved this issue in our Staging environment.

As there is no documentation on ServiceProviderMode (that I can find ...) can you offer some guidance on the difference between the two modes?

I can see that the default Dynamic mode is compiled whereas runtime is resolved at request, am I correct in thinking we should only be expecting a small performance hit by switching mode until a fix is released?

[analogrelay]

You'd have to measure in your scenario to know for sure. Here's a brief summary of the modes:

• Runtime means that every time a service needs to be created, we basically use Activator.CreateInstance to do so. For singletons, this has minimal impact since they are created once per app. For transient and scoped services, reflection is much slower than directly calling new, so it has a noticible impact
• Dynamic means that the first time a service needs to be created, we compile IL into a new DynamicMethod that will actually use new (or rather than newobj IL opcode ;)) to create an instance of the type. We then store that delegate and use it to construct instances each time. It's still a little slower than directly calling new but it's much faster than constructing the instance using reflection each time.

So I would expect some performance impact from switching modes.

[AdamRiddick]

Hi Andrew,

I can confirm this was the route cause of the high CPU on startup & the OutOfMemoryException's we were seeing throughout the day when the application was starting up again.

We are now averaging a slightly higher CPU usage, but thankfully it's nothing Drastic.

We'll keep an eye on #2737 relating to this - Thanks.

[analogrelay]

Using the Dynamic mode was the root cause? Do you have a large number of DI services? I believe you can query the count of the IServiceCollection at the end of your ConfigureServices to check this.

The fact that switching dramatically reduces CPU/Memory load indicates a very large number of DI services, all of which are used at startup, meaning we have to compile a lot of these dynamic methods.

[gregpakes]

@anurse

I’m going to reply on behalf of @AdamRiddick (same company). I believe Adam is on leave.

No we do not have a large number of DI services. The application is quite small. It is a custom version of Identity Server 4.

The more we look at it it looks like a bug/issue with the dynamic mode.

Did you manage to look at the dump Adam sent? That shows the issue quite well.

[analogrelay]

Interesting. I did look at the dump and responded indicating that it seemed to be missing data (my bad, this message didn't get through). I can try to take another look.

[analogrelay]

Yeah, I double checked the dump and it looks like it may be after the memory spike is cleaned up. For example, I see the total GC heap size reported as only about ~130MB:

0:000> !EEHeap -gc
Number of GC Heaps: 1
generation 0 starts at 0x0000016886f93a60
generation 1 starts at 0x0000016886aeea88
generation 2 starts at 0x0000016880001000
ephemeral segment allocation context: none
         segment             begin         allocated              size
0000016880000000  0000016880001000  00000168873e45d0  0x73e35d0(121517520)
Large object heap starts at 0x0000016890001000
         segment             begin         allocated              size
0000016890000000  0000016890001000  00000168915498b0  0x15488b0(22317232)
Total Size:              Size: 0x892be80 (143834752) bytes.
------------------------------
GC Heap Size:            Size: 0x892be80 (143834752) bytes.

I do see 4GB of Private Bytes recorded in the dump (the MEM_PRIVATE category in your results above), but it appears the dump occurred after a garbage collection but before the memory was returned to the OS so I'm not really able to see what is allocating the large amounts of memory.

Did you find more relevant data in the dump? That might help point me in the right direction. Is it possible for you to share the application code at all?