Linkerd proxy randomly panics after linkerd-desination OOM

[ketchoop]

What is the issue?

Experiencing connection resets to some pods in cluster from pod with injected linkerd-proxy. Restarting problematic pod that can appear after some time solves the issue. Restarting all pods of deployment fixes issue completely. Linkerd-destination is up and ready when the issue is happening and looks like linkerd-proxy can't self recover after this.

How can it be reproduced?

Restarting linkerd destination several times may help to reproduce

Logs, error output, etc

[1479104.724995s]  WARN ThreadId(01) outbound: linkerd_app_core::serve: Server failed to become ready error=buffer's worker closed unexpectedly client.addr=10.x.x.x:xxxx
thread 'main' panicked at /__w/linkerd2-proxy/linkerd2-proxy/linkerd/proxy/balance/queue/src/service.rs:73:18:
worker must set a failure if it exits prematurely
[1479362.242617s]  WARN ThreadId(01) outbound: linkerd_app_core::serve: Server failed to become ready error=buffer's worker closed unexpectedly client.addr=10.x.x.x:xxxx
thread 'main' panicked at /__w/linkerd2-proxy/linkerd2-proxy/linkerd/proxy/balance/queue/src/service.rs:73:18:
worker must set a failure if it exits prematurely
[1479766.042212s]  WARN ThreadId(01) outbound: linkerd_app_core::serve: Server failed to become ready error=buffer's worker closed unexpectedly client.addr=10.x.x.x:x:xxxx
thread 'main' panicked at /__w/linkerd2-proxy/linkerd2-proxy/linkerd/proxy/balance/queue/src/service.rs:73:18:
worker must set a failure if it exits prematurely
[1480021.068217s]  WARN ThreadId(01) outbound: linkerd_app_core::serve: Server failed to become ready error=buffer's worker closed unexpectedly client.addr=10.x.x.x:x

thread 'main' panicked at linkerd/app/core/src/control.rs:116:49:

output of linkerd check -o short

Status check results are √

Environment

Linkerd version: 25.2.1
Kubernetes cluster: v1.30.13

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None

[olix0r]

The referenced code is:

https://github.com/linkerd/linkerd2-proxy/blob/b2e8623419eebe8ba8a52e6c7adc85e9a836960b/linkerd/app/core/src/control.rs#L116

Were the panics preceded by these warnings? If so, the details of error message may be helpful:

https://github.com/linkerd/linkerd2-proxy/blob/b2e8623419eebe8ba8a52e6c7adc85e9a836960b/linkerd/app/core/src/control.rs#L112

It does appear that the interval call includes an assertion internally:

    assert!(period > Duration::new(0, 0), "`period` must be non-zero.");

Did that error message show up in the logs?

Regardless, we should probably proactively defend against that condition.

[ketchoop]

Were the panics preceded by these warnings?

Honestly, not sure(and can't confirm this because logs are gone), but it can be. Because we've reconstructed the entire incident timeline and have info that the Linkerd-destination was under heavy load because of restarting pods and moreover we had our local dns under pressure(we know that there is a bug with linkerd-proxy and dns resolution that causes huge load to dns).

So, it's possible that proxies couldn't resolve/connect to linkerd proxy and after that we've got panic.

Did that error message show up in the logs?

No, I haven't seen this message before.