linkerd-service-mirror fails on patching link status when mirrored service namespace is not present #14433
Description
What is the issue?
This has been observed on 3 k8s clusters with pod-to-pod multi-cluster communication.
On one cluster we have a service foo in namespace bar. The service is annotated for mirroring with mirror.linkerd.io/mirrored-service: "true" annotation.
On another cluster namespace bar is not present.
linkerd-service-mirror on that cluster logs:
level=warning msg="Skipping mirroring of service bar/foo: namespace bar does not exist" cluster=c1
level=info msg="patching link status linkerd-multicluster/c1" cluster=c1
level=error msg="Failed to patch link status linkerd-multicluster/c1: [Link.multicluster.linkerd.io](http://link.multicluster.linkerd.io/) \"c1\" is invalid: [status.federatedServices[1].conditions[0].localRef.name: Required value, status.federatedServices[1].conditions[0].localRef.namespace: Required value]" cluster=c1
Interestingly status.federatedServices of c1 link has only one element.
I would expect that if the namespace does not exist then the link status shouldn't be patched (and log error).
How can it be reproduced?
3 k8s clusters (1.33) with pod-to-pod multi-cluster communication.
linkerd-service-mirror-* deployments - image: cr.l5d.io/linkerd/controller:edge-25.2.1
the rest of the linkerd stack in version edge-25.7.1
on one cluster mirrored service in a namespace that is not present on other cluster
Logs, error output, etc
level=warning msg="Skipping mirroring of service bar/foo: namespace bar does not exist" cluster=c1
level=info msg="patching link status linkerd-multicluster/c1" cluster=c1
level=error msg="Failed to patch link status linkerd-multicluster/c1: [Link.multicluster.linkerd.io](http://link.multicluster.linkerd.io/) \"c1\" is invalid: [status.federatedServices[1].conditions[0].localRef.name: Required value, status.federatedServices[1].conditions[0].localRef.namespace: Required value]" cluster=c1
output of linkerd check -o short
linkerd-webhooks-and-apisvc-tls
-------------------------------
‼ proxy-injector cert is valid for at least 60 days
certificate will expire on 2025-09-03T01:31:21Z
see https://linkerd.io/2/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
‼ sp-validator cert is valid for at least 60 days
certificate will expire on 2025-09-03T01:31:22Z
see https://linkerd.io/2/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints
‼ policy-validator cert is valid for at least 60 days
certificate will expire on 2025-09-03T01:31:21Z
see https://linkerd.io/2/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints
linkerd-version
---------------
‼ cli is up-to-date
is running version 25.8.4 but the latest edge version is 25.8.5
see https://linkerd.io/2/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
is running version 25.7.1 but the latest edge version is 25.8.5
see https://linkerd.io/2/checks/#l5d-version-control for hints
‼ control plane and cli versions match
control plane running edge-25.7.1 but cli running edge-25.8.4
see https://linkerd.io/2/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-7c44669787-9j87z (edge-25.7.1)
* linkerd-destination-7c44669787-rjzzm (edge-25.7.1)
* linkerd-destination-7c44669787-zqg6c (edge-25.7.1)
* linkerd-identity-6b7dc5555-4cngb (edge-25.7.1)
* linkerd-identity-6b7dc5555-6dm8z (edge-25.7.1)
* linkerd-identity-6b7dc5555-l9b46 (edge-25.7.1)
* linkerd-proxy-injector-b6776c8f-g68b7 (edge-25.7.1)
* linkerd-proxy-injector-b6776c8f-ghwdc (edge-25.7.1)
* linkerd-proxy-injector-b6776c8f-gp2x8 (edge-25.7.1)
see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
linkerd-destination-7c44669787-9j87z running edge-25.7.1 but cli running edge-25.8.4
see https://linkerd.io/2/checks/#l5d-cp-proxy-cli-version for hints
linkerd-multicluster
--------------------
‼ Link and CLI versions match
* c2: CLI version is edge-25.8.4 but Link version is edge-25.2.1
* c1: CLI version is edge-25.8.4 but Link version is edge-25.2.1
see https://linkerd.io/2/checks/#l5d-multicluster-links-version for hints
‼ extension is managing controllers
* using legacy service mirror controller for Link: c2
* using legacy service mirror controller for Link: c1
see https://linkerd.io/2/checks/#l5d-multicluster-managed-controllers for hints
‼ multicluster extension proxies are up-to-date
some proxies are not running the current version:
* linkerd-local-service-mirror-76d849fccc-wjxlb (edge-25.7.1)
* linkerd-service-mirror-c2-b85848984-qjdd5 (edge-25.7.1)
* linkerd-service-mirror-c1-78b57764f6-txt2x (edge-25.7.1)
* linkerd-local-service-mirror-76d849fccc-wjxlb (edge-25.7.1)
* linkerd-service-mirror-c2-b85848984-qjdd5 (edge-25.7.1)
* linkerd-service-mirror-c1-78b57764f6-txt2x (edge-25.7.1)
see https://linkerd.io/2/checks/#l5d-multicluster-proxy-cp-version for hints
‼ multicluster extension proxies and cli versions match
linkerd-local-service-mirror-76d849fccc-wjxlb running edge-25.7.1 but cli running edge-25.8.4
see https://linkerd.io/2/checks/#l5d-multicluster-proxy-cli-version for hints
linkerd-viz
-----------
‼ tap API server cert is valid for at least 60 days
certificate will expire on 2025-09-03T01:31:18Z
see https://linkerd.io/2/checks/#l5d-tap-cert-not-expiring-soon for hints
‼ viz extension proxies are up-to-date
some proxies are not running the current version:
* metrics-api-cb74b89bf-7lpdr (edge-25.7.1)
* tap-c7f999d85-s97b6 (edge-25.7.1)
* tap-injector-85ff99c7d6-zr7fn (edge-25.7.1)
* web-6897c8b96c-5pgxk (edge-25.7.1)
see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
metrics-api-cb74b89bf-7lpdr running edge-25.7.1 but cli running edge-25.8.4
see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints
Status check results are √
Environment
- Kubernetes 1.33
- Amazon EKS
- Bottlerocket OS 1.45.0 (aws-k8s-1.33)
Possible solution
No response
Additional context
No response
Would you like to work on fixing this bug?
None