Use `GOMEMLIMIT` to guard against OOM kills in controllers - Upgrade go to `1.19`

[alpeb]

Problem and repro

In some situations the destination container sees its memory consumption progressively augment until it gets OOM killed. We have been able to reproduce that by rolling out emojivoto's pods every 30s (here the rollouts started at 11:55):

(tracking here container_memory_working_set_bytes{container="destination"}, which is the metric the OOM killer watches)

We took heap dumps before starting the rollouts and 25m after, and generated this comparative result:

# (using edge-22.12.1)
$ k -n linkerd port-forward linkerd-destination-5dd6c5985f-sl2jz 9996 &
$ curl http://localhost:9996/debug/pprof/heap?gc=1 > heap-before.out
$ while true; do kubectl -n emojivoto rollout restart deploy ; sleep 30; done
# wait 10m
$ curl http://localhost:9996/debug/pprof/heap?gc=1 > heap-after.out
$ go tool pprof -png --base heap-before.out heap-after.out

This shows 68.71% of the allocated memory during this time span was requested by k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive, so not linkerd's fault.

GC tuning

Lowering GOGC down to 1 forces GC to run as frequently as possible, but the curve above just smoothens up while overall memory consumption and trend remains the same.

However, using GOMEMLIMIT (introduced in go 1.19) helps out. If we set it to 35MiB we get this, triggering the rollouts at 14:32:

Before, we went from 26MB up to 42MB during the rollouts, and now from 25MB to 26MB in about the same time. Memory usage still climbs, but not as bad as before.

GOMEMLIMIT requires go 1.19, and it's actually recommended to use in containers with a memory limit; according to A Guide to the Go Garbage Collector:

Do take advantage of the memory limit when the execution environment of your Go program is entirely within your control, and the Go program is the only program with access to some set of resources (i.e. some kind of memory reservation, like a container memory limit).
A good example is the deployment of a web service into containers with a fixed amount of available memory.
In this case, a good rule of thumb is to leave an additional 5-10% of headroom to account for memory sources the Go runtime is unaware of.

Course of action

Besides bumping go to 1.19 in all the base images, we can add the env var GOMEMLIMIT to the manifests only if the helm value memory.limit exists (values-ha.yaml sets it for the control plane containers), set to 90% its value.

Ref #8270

[olix0r]

This shows 68.71% of the allocated memory during this time span was requested by k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive, so not linkerd's fault.

I don't draw that conclusion from this data.

Yes, 69% of allocations occur when parsing JSON data from Kubernetes. This isn't really shocking for an application that mostly watches lots of resources. Pprof doesn't tell us how long these allocations last, though. They could be extremely short-lived allocations, in which case, who cares? Or the parsed objects could be held forever (i.e. by Linkerd), which would actually be a leak. Or, it could just as well be that something in the other 31% of allocations is being retained in a way that appears leaky.

pprof is useful to help us fill in our picture of the system, but we should be wary about drawing conclusions based on it alone.

If GOMEMLIMIT is something we should be setting in any case, it seems like a good change (so that the runtime knows its actual available memory). But I'm still pretty skeptical that this explains/resolves the underlying issue: while it's good that we can increase GC as memory pressure increases, and this helps us avoid OOMing, we probably don't want to be in that state in general (in fact it seems like something worth alerting on -- a destination controller in that state should probably get more memory so that it doesn't suffer continuous GC).

What we're really trying to understand is: why do we have memory pressure in the first place? Is it really JSON decoding? If so, it seems like the Kubernetes API server (as well as numerous other controllers) would suffer from this as well. What have they done to address this?

How much memory do we need to allocate to avoid this GC pressure?

[olix0r]

Does the rate of JSON allocations increase over time? It seems like it should be basically flat/predictable (and that GC should happily clear old data when it runs). If it's flat, that doesn't really explain the behavior we were seeing. If it's increasing, that would be a very interesting lead to chase down...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[Teko012]

We still see regular OOM kills in linkerd-proxy, was this implemented for that as well?

[kleimkuhler]

@Teko012 what version of Linkerd are you running? We've had some memory fixes in the recent edges so I'm curious if you still see those on the newer versions.

[Teko012]

@kleimkuhler We are on the latest stable 2.12.4.

[kleimkuhler]

Okay it'd be helpful if you can open up a separate issue with a description about what you're seeing and any relevant information. I'd like to keep this issue specific to the ongoing conversation and if GOMEMLIMIT is worth adding, but we can meanwhile dig a little more into your issue.