Linkerd is not working with linkerd-cni enabled #14538
Description
What is the issue?
Linkerd pods are not starting when Linkerd is installed with CNI enabled.
How can it be reproduced?
linkerd install-cni | kubectl apply -f -
linkerd install --crds | kubectl apply -f -
linkerd install --linkerd-cni-enabled | kubectl apply -f -
Logs, error output, etc
Warning FailedCreatePodSandBox 25m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5f5e43c2bbc1bd89b8604705c2eda076e00593fe4c26f8ace6cf955b3bdcad0e": plugin type="linkerd-cni" name="linkerd-cni" failed (add): exit status 127
Warning FailedCreatePodSandBox 24m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "69c58b4c31916475fcc35908d9b8a8c2ef5937d6acb8d8cbe6aa8eaf4e281e11": plugin type="linkerd-cni" name="linkerd-cni" failed (add): exit status 127
output of linkerd check -o short
linkerd-existence
-----------------
Environment
- Kubernetes Version: v1.33
- Cluster Environment: EKS
- Host OS: Bottlerocket OS 1.47.0 (aws-k8s-1.33)
- Linkerd version: edge-25.8.5
Possible solution
No response
Additional context
We are using AWS CNI, linkerd-cni pods are running, no errors.
[2025-09-26 09:03:08] Wrote linkerd CNI binaries to /host/opt/cni/bin
Setting up watches.
Watches established.
[2025-09-26 09:03:08] Trigger CNI config detection for /host/etc/cni/net.d/10-aws.conflist
Setting up watches.
Watches established.
[2025-09-26 09:03:08] Detected event: CREATE /host/etc/cni/net.d/10-aws.conflist
[2025-09-26 09:03:08] New/changed file [/host/etc/cni/net.d/10-aws.conflist] detected; re-installing
[2025-09-26 09:03:08] Using CNI config template from CNI_NETWORK_CONFIG environment variable.
[2025-09-26 09:03:08] CNI config: {
"name": "linkerd-cni",
"type": "linkerd-cni",
"log_level": "info",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/ZZZ-linkerd-cni-kubeconfig"
},
"linkerd": {
"incoming-proxy-port": 4143,
"outgoing-proxy-port": 4140,
"proxy-uid": 2102,
"ports-to-redirect": [],
"inbound-ports-to-ignore": ["4191","4190"],
"simulate": false,
"use-wait-flag": false,
"iptables-mode": "nft",
"ipv6": false
}
}
[2025-09-26 09:03:08] Created CNI config /host/etc/cni/net.d/10-aws.conflist
[2025-09-26 09:03:08] Detected event: MODIFY /host/etc/cni/net.d/10-aws.conflist
[2025-09-26 09:03:08] Ignoring event: MODIFY /host/etc/cni/net.d/10-aws.conflist; no real changes detected or file disappeared
[2025-09-26 09:03:08] Detected event: CREATE /host/etc/cni/net.d/10-aws.conflist
[2025-09-26 09:03:08] Ignoring event: CREATE /host/etc/cni/net.d/10-aws.conflist; no real changes detected or file disappeared
[2025-09-26 09:03:08] Detected event: MODIFY /host/etc/cni/net.d/10-aws.conflist
[2025-09-26 09:03:08] Ignoring event: MODIFY /host/etc/cni/net.d/10-aws.conflist; no real changes detected or file disappeared
[2025-09-26 09:52:12] Detected change in service account files; recreating kubeconfig file
When I login to Bottlerocket node, linkerd-cni binary is where expected /opt/cni/bin and is executable with proper flags like aws-cni.
bash-5.1# ls -Z /opt/cni/bin/aws-cni
system_u:object_r:cni_exec_t:s0 /opt/cni/bin/aws-cni
bash-5.1# ls -Z /opt/cni/bin/linkerd-cni
system_u:object_r:cni_exec_t:s0 /opt/cni/bin/linkerd-cni
bash-5.1# cat /etc/cni/net.d/10-aws.conflist
{
"cniVersion": "0.4.0",
"name": "aws-cni",
"disableCheck": true,
"plugins": [
{
"name": "aws-cni",
"type": "aws-cni",
"vethPrefix": "eni",
"mtu": "9001",
"podSGEnforcingMode": "standard",
"pluginLogFile": "/var/log/aws-routed-eni/plugin.log",
"pluginLogLevel": "DEBUG",
"capabilities": {
"io.kubernetes.cri.pod-annotations": true
}
},
{
"name": "egress-cni",
"type": "egress-cni",
"mtu": "9001",
"enabled": "false",
"randomizeSNAT": "prng",
"nodeIP": "",
"ipam": {
"type": "host-local",
"ranges": [
[
{
"subnet": "fd00::ac:00/118"
}
]
],
"routes": [
{
"dst": "::/0"
}
],
"dataDir": "/run/cni/v4pd/egress-v6-ipam"
},
"pluginLogFile": "/var/log/aws-routed-eni/egress-v6-plugin.log",
"pluginLogLevel": "DEBUG"
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
},
"snat": true
},
{
"name": "linkerd-cni",
"type": "linkerd-cni",
"log_level": "info",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/ZZZ-linkerd-cni-kubeconfig"
},
"linkerd": {
"incoming-proxy-port": 4143,
"outgoing-proxy-port": 4140,
"proxy-uid": 2102,
"ports-to-redirect": [],
"inbound-ports-to-ignore": [
"4191",
"4190"
],
"simulate": false,
"use-wait-flag": false,
"iptables-mode": "nft",
"ipv6": false
}
}
]
}
AWS CNI Logs
{"level":"info","ts":"2025-09-26T10:35:10.511Z","caller":"routed-eni-cni-plugin/cni.go:133","msg":"Constructed new logger instance"}
{"level":"info","ts":"2025-09-26T10:35:10.511Z","caller":"routed-eni-cni-plugin/cni.go:142","msg":"Received CNI add request: ContainerID(c4dd4b3544cc3a6fda567e7f3654140bfdadefcff5178280800c27b1d55da424) Netns(/var/run/netns/cni-6473ab7d-a969-5c50-aee5-f6b4ac0c1f38) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=linkerd;K8S_POD_NAME=linkerd-identity-867f484856-tc2gp;K8S_POD_INFRA_CONTAINER_ID=c4dd4b3544cc3a6fda567e7f3654140bfdadefcff5178280800c27b1d55da424;K8S_POD_UID=0c5dcff2-b098-4d58-a675-75cd08ffddff) Path(/opt/cni/bin) argsStdinData({\"capabilities\":{\"io.kubernetes.cri.pod-annotations\":true},\"cniVersion\":\"0.4.0\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"podSGEnforcingMode\":\"standard\",\"runtimeConfig\":{\"io.kubernetes.cri.pod-annotations\":{\"cluster-autoscaler.kubernetes.io/safe-to-evict\":\"true\",\"config.linkerd.io/default-inbound-policy\":\"all-unauthenticated\",\"kubernetes.io/config.seen\":\"2025-09-26T09:08:29.314351775Z\",\"kubernetes.io/config.source\":\"api\",\"linkerd.io/created-by\":\"linkerd/cli edge-25.8.5\",\"linkerd.io/proxy-version\":\"edge-25.8.5\",\"linkerd.io/trust-root-sha256\":\"94bc45921ee0b5d2d4352cfdd7c9219b44fccdb9adca2880fce7265a12ca969d\"}},\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"debug","ts":"2025-09-26T10:35:10.511Z","caller":"routed-eni-cni-plugin/cni.go:142","msg":"Prev Result: <nil>\n"}
{"level":"debug","ts":"2025-09-26T10:35:10.511Z","caller":"routed-eni-cni-plugin/cni.go:142","msg":"MTU value set is 9001:"}
{"level":"debug","ts":"2025-09-26T10:35:10.511Z","caller":"routed-eni-cni-plugin/cni.go:142","msg":"pod requires multi-nic attachment: false"}
{"level":"info","ts":"2025-09-26T10:35:10.514Z","caller":"routed-eni-cni-plugin/cni.go:142","msg":"Received add network response from ipamd for container c4dd4b3544cc3a6fda567e7f3654140bfdadefcff5178280800c27b1d55da424 interface eth0: Success:true IPAllocationMetadata:{IPv4Addr:\"10.132.55.205\" DeviceNumber:1 RouteTableId:2} VPCv4CIDRs:\"10.132.0.0/16\" NetworkPolicyMode:\"standard\""}
{"level":"debug","ts":"2025-09-26T10:35:10.514Z","caller":"routed-eni-cni-plugin/cni.go:281","msg":"SetupPodNetwork: hostVethName=enic10159c4e47, contVethName=eth0, netnsPath=/var/run/netns/cni-6473ab7d-a969-5c50-aee5-f6b4ac0c1f38, ipAddr=10.132.55.205/32, routeTableNumber=2, mtu=9001"}
{"level":"debug","ts":"2025-09-26T10:35:10.562Z","caller":"driver/driver.go:276","msg":"Successfully set IPv6 sysctls on hostVeth enic10159c4e47"}
{"level":"debug","ts":"2025-09-26T10:35:10.563Z","caller":"driver/driver.go:286","msg":"Successfully setup container route, containerAddr=10.132.55.205/32, hostVeth=enic10159c4e47, rtTable=main"}
{"level":"debug","ts":"2025-09-26T10:35:10.563Z","caller":"driver/driver.go:286","msg":"Successfully setup toContainer rule, containerAddr=10.132.55.205/32, rtTable=main"}
{"level":"debug","ts":"2025-09-26T10:35:10.563Z","caller":"driver/driver.go:286","msg":"Successfully setup fromContainer rule, containerAddr=10.132.55.205/32, rtTable=2"}
{"level":"debug","ts":"2025-09-26T10:35:10.563Z","caller":"routed-eni-cni-plugin/cni.go:142","msg":"Using dummy interface: {Name:dummyc10159c4e47 Mac:0 Mtu:0 Sandbox:1 SocketPath: PciID:}"}
{"level":"debug","ts":"2025-09-26T10:35:10.564Z","caller":"routed-eni-cni-plugin/cni.go:142","msg":"Network Policy agent for EnforceNpToPod returned Success : true"}
{"level":"info","ts":"2025-09-26T10:35:10.653Z","caller":"routed-eni-cni-plugin/cni.go:133","msg":"Constructed new logger instance"}
{"level":"debug","ts":"2025-09-26T10:35:10.654Z","caller":"routed-eni-cni-plugin/cni.go:363","msg":"Prev Result: <nil>\n"}
{"level":"info","ts":"2025-09-26T10:35:10.654Z","caller":"routed-eni-cni-plugin/cni.go:363","msg":"Received CNI del request: ContainerID(c4dd4b3544cc3a6fda567e7f3654140bfdadefcff5178280800c27b1d55da424) Netns(/var/run/netns/cni-6473ab7d-a969-5c50-aee5-f6b4ac0c1f38) IfName(eth0) Args(K8S_POD_UID=0c5dcff2-b098-4d58-a675-75cd08ffddff;IgnoreUnknown=1;K8S_POD_NAMESPACE=linkerd;K8S_POD_NAME=linkerd-identity-867f484856-tc2gp;K8S_POD_INFRA_CONTAINER_ID=c4dd4b3544cc3a6fda567e7f3654140bfdadefcff5178280800c27b1d55da424) Path(/opt/cni/bin) argsStdinData({\"capabilities\":{\"io.kubernetes.cri.pod-annotations\":true},\"cniVersion\":\"0.4.0\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"podSGEnforcingMode\":\"standard\",\"runtimeConfig\":{\"io.kubernetes.cri.pod-annotations\":{\"cluster-autoscaler.kubernetes.io/safe-to-evict\":\"true\",\"config.linkerd.io/default-inbound-policy\":\"all-unauthenticated\",\"kubernetes.io/config.seen\":\"2025-09-26T09:08:29.314351775Z\",\"kubernetes.io/config.source\":\"api\",\"linkerd.io/created-by\":\"linkerd/cli edge-25.8.5\",\"linkerd.io/proxy-version\":\"edge-25.8.5\",\"linkerd.io/trust-root-sha256\":\"94bc45921ee0b5d2d4352cfdd7c9219b44fccdb9adca2880fce7265a12ca969d\"}},\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"info","ts":"2025-09-26T10:35:10.655Z","caller":"routed-eni-cni-plugin/cni.go:363","msg":"Received del network response from ipamd for pod linkerd-identity-867f484856-tc2gp namespace linkerd sandbox c4dd4b3544cc3a6fda567e7f3654140bfdadefcff5178280800c27b1d55da424: Success:true IPAllocationMetadata:{IPv4Addr:\"10.132.55.205\" DeviceNumber:1 RouteTableId:2} NetworkPolicyMode:\"standard\""}
{"level":"debug","ts":"2025-09-26T10:35:10.655Z","caller":"routed-eni-cni-plugin/cni.go:489","msg":"TeardownPodNetwork: containerAddr=10.132.55.205/32, routeTable=2"}
{"level":"debug","ts":"2025-09-26T10:35:10.656Z","caller":"driver/driver.go:307","msg":"Successfully deleted toContainer rule, containerAddr=10.132.55.205/32, rtTable=main"}
{"level":"debug","ts":"2025-09-26T10:35:10.656Z","caller":"driver/driver.go:307","msg":"Successfully deleted fromContainer rule, containerAddr=10.132.55.205/32, rtTable=2"}
{"level":"debug","ts":"2025-09-26T10:35:10.656Z","caller":"driver/driver.go:307","msg":"Successfully deleted container route, containerAddr=10.132.55.205/32, rtTable=main"}
{"level":"debug","ts":"2025-09-26T10:35:10.657Z","caller":"routed-eni-cni-plugin/cni.go:363","msg":"Network Policy agent for DeletePodNp returned Success : true"}
Would you like to work on fixing this bug?
None