What can we do when an API consume to much memory?

[waldo2188]

Describe the bug

We run some ASP .Net Core REST API with Docker and Docker-compose on Linux Debian. Some time one of this API start to increase his use of memory, because the API need to run a heavy process. But when the process is done, the memory never decreases.

To Reproduce

So, I’ve started a new API (ASP Net Core 2.2) from scratch to understand what’s wrong https://github.com/waldo2188/PleaseDontCrash.

This API use App Metrics, Prometheus and Grafana.

The API is simple:

• http://localhost:5000/api/values : is an entry point made to increase memory usage.
• http://localhost:3000 : is for Grafana, you can use the file Please_Don_t_Crash-grafana.json as dashboard.

The tests describe below have been made on Windows 10 with Docker for Windows on a Linux base.
First, I start the API and the Physical Memory Usage is about 90Mb.

After 20min with no action on the API the Physical Memory Usage is about 162Mb.

I run the URL localhost:5000/api/values 10 time to increase memory, and when the HTTP request in done, the Physical Memory Usage is about 2400Mb and thee Garbage Collector is about 2300Mb.

After 20 min with no action on the API the Physical Memory Usage is about 2456Mb and thee Garbage Collector is about 2398Mb.

After 40 min with no action on the API the Physical Memory Usage is about 2492Mb and thee Garbage Collector is about 2445Mb.

After 60 min with no action on the API the Physical Memory Usage is about 2566Mb and thee Garbage Collector is about 2543Mb.

What can we do to reduce memory consumption when the API is clearly not being used?

Expected behaviour

We expect that the Garbage Collector reduce de memory consumption.

Screenshots

Grafana_Please_Don_t_Crash.png

[davidfowl]

This may help https://blogs.msdn.microsoft.com/maoni/2018/10/02/middle-ground-between-server-and-workstation-gc/ and this https://blogs.msdn.microsoft.com/maoni/2018/11/16/running-with-server-gc-in-a-small-container-scenario-part-0/

[waldo2188]

Thank you for the links. They help to understand what is happening, but not to solve it.

[Eilon]

@sebastienros - any additional thoughts on how to diagnose this?

[guneysus]

We are having this problem on Aspnet Core 1. We were able to catch the moment and took some memory dumps when API starts to hog server RAM by 100MB/s.

But memory dump analyses does not show the real cause of problem.

You can see the our methodology to approach to find root cause here: #6167

[sebastienros]

We expect that the Garbage Collector reduce de memory consumption.

What you expect is not what the GC does, and it's not a bug, it does it by design, so from a dotnet runtime perspective there is no issue here, just expected behavior

To "solve" this you would need to accept either of these facts:
a- the GC uses as much memory as it can if there is no memory pressure on the system
b- use the workstation GC to limit how much memory the GC is using, at the cost of performance

However you are not the first one to mention that this behavior is not optimal, and many users would love to have a mode where memory can be constrained, even more in the context of micro-services where many asp.net processes can run on servers that have a lot of cores and available memory. I believe the GC team is already working on such features.