[Bug]: Concurrent ComputeDomain status updates can lose peer entries and leave the domain non-converged

[aditighag]

Component

compute-domain-daemon

Bug Description

When multiple compute-domain daemon pods update ComputeDomain.Status.Nodes concurrently, each pod rewrites the full status.nodes from its own version of the CD object. This can cause lost node updates, and leave a ComputeDomain that stays NotReady even after expected number of nodes have joined.

Snippets from real cluster highlighting the issue -

Status:
Nodes:
Clique Id: 9f410827-489f-46fd-8203-51266ca08eb7.32766
Index: 0
Ip Address: 10.42.3.138
Name: cluster1-mgx-00011
Status: NotReady
Clique Id: 9f410827-489f-46fd-8203-51266ca08eb7.32766
Index: 1
Ip Address: 10.42.1.100
Name: cluster1-mgx-00017
Status: NotReady
Status: NotReady
Events:

I0415 16:14:28.038085 1 cdstatus.go:276] Successfully inserted/updated node in CD (nodeinfo: &{cluster1-mgx-00011 10.42.3.138 9f410827-489f-46fd-8203-51266ca08eb7.32766 0 NotReady})
I0415 16:14:28.038098 1 cdstatus.go:339] numNodes: 2, nodes seen: 1
I0415 16:14:28.044285 1 cdstatus.go:239] syncNodeInfoToCD noop: pod IP unchanged (10.42.3.138)
I0415 16:14:28.044300 1 cdstatus.go:339] numNodes: 2, nodes seen: 1
I0415 16:14:28.053413 1 cdstatus.go:239] syncNodeInfoToCD noop: pod IP unchanged (10.42.3.138)
I0415 16:14:28.053424 1 cdstatus.go:354] IP set for clique did not change

I0415 16:14:30.033811 1 cdstatus.go:259] CD status does not contain node name 'cluster1-mgx-00011' yet, try to insert myself: &{cluster1-mgx-00011 9f410827-489f-46fd-8203-51266ca08eb7.32766 0 NotReady}
I0415 16:14:30.037427 1 round_trippers.go:632] "Response" verb="PUT" url="https://10.43.0.1:443/apis/resource.nvidia.com/v1beta1/namespaces/default/computedomains/imex-channel-injection/status" status="200 OK" milliseconds=3
I0415 16:14:30.037620 1 cdstatus.go:276] Successfully inserted/updated node in CD (nodeinfo: &{cluster1-mgx-00011 10.42.3.138 9f410827-489f-46fd-8203-51266ca08eb7.32766 0 NotReady})
I0415 16:14:30.037656 1 cdstatus.go:339] numNodes: 2, nodes seen: 1
I0415 16:14:30.042979 1 cdstatus.go:239] syncNodeInfoToCD noop: pod IP unchanged (10.42.3.138)
I0415 16:14:30.042995 1 cdstatus.go:339] numNodes: 2, nodes seen: 1
I0415 16:14:30.052870 1 cdstatus.go:239] syncNodeInfoToCD noop: pod IP unchanged (10.42.3.138)
I0415 16:14:30.052888 1 cdstatus.go:354] IP set for clique did not change

Here is a unit test to reproduce the issue - #1049.

Steps to Reproduce

1. Deploy with helm
   helm upgrade -i nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu --namespace nvidia-dra-driver-gpu --create-namespace --set nvidiaDriverRoot=/ --set gpuResourcesEnabledOverride=true --version="25.12.0”

Disable IMEXDaemonsWithDNSNames (and ComputeDomainCliques) feature flags.

2. Deploy a workload requesting computedomain resource with multiple nodes (> 1).
   Able to reproduce the issue with a modified version of the imex-channel-injection.yaml (only used for testing) where numNodes field is updated to 2 and also the mpi operator workload listed on the validation page.

---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: imex-channel-injection
spec:
  numNodes: 2
  channel:
    allocationMode: All
    resourceClaimTemplate:
      name: imex-channel-0
---
apiVersion: v1
kind: Pod
metadata:
  name: imex-channel-injection
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.clique
            operator: Exists
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["ls -la /dev/nvidia-caps-imex-channels; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: imex-channel-0
  resourceClaims:
  - name: imex-channel-0
    resourceClaimTemplateName: imex-channel-0
---
apiVersion: v1
kind: Pod
metadata:
  name: imex-channel-injection2
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.clique
            operator: Exists
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["ls -la /dev/nvidia-caps-imex-channels; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: imex-channel-0
  resourceClaims:
  - name: imex-channel-0
    resourceClaimTemplateName: imex-channel-0

Expected Behavior

ComputeDomain moves to Ready state once expected number of nodes are joined. Example output when ComputeDomainCliques is enabled -

$ k describe computedomain nvbandwidth-test-compute-domain
Name:         nvbandwidth-test-compute-domain
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  [[resource.nvidia.com/v1beta1](http://resource.nvidia.com/v1beta1)](http://resource.nvidia.com/v1beta1)
Kind:         ComputeDomain
Metadata:
Creation Timestamp:  2026-04-15T22:46:21Z
Finalizers:
[[resource.nvidia.com/computeDomain](http://resource.nvidia.com/computeDomain)](http://resource.nvidia.com/computeDomain)
Generation:        1
Resource Version:  4221096
UID:               d7245c1a-3bf6-460b-90a3-d8ed8bbf12d4
Spec:
Channel:
Allocation Mode:  Single
Resource Claim Template:
Name:   nvbandwidth-test-compute-domain-channel
Num Nodes:  2
Status:
Nodes:
Clique Id:   9f410827-489f-46fd-8203-51266ca08eb7.32766
Index:       0
Ip Address:  10.42.1.165
Name:        cluster1-mgx-00017
Status:      Ready
Clique Id:   9f410827-489f-46fd-8203-51266ca08eb7.32766
Index:       1
Ip Address:  10.42.0.69
Name:        cluster1-mgx-00010
Status:      Ready
Status:        Ready
Events:          <none>

DRA Driver Version

v25.12.0

Kubernetes Version

v1.35 rke

GPU Model

NVIDIA GB300

NVIDIA Driver Version

No response

OS / Kernel

No response

Container Runtime

No response

Feature Gates (non-default settings)

IMEXDaemonsWithDNSNames (disabled)

Helm Values (non-default)

Relevant Logs

Debug Information Attached

• kubectl get pods -n dra-driver-nvidia-gpu -o wide
• kubectl get resourceclaims -n <namespace>
• kubectl get resourceslices.resource.k8s.io
• kubectl describe pod <pod> or kubectl events --for pod/<pod>
• Kubelet plugin logs: kubectl logs -n dra-driver-nvidia-gpu -l dra-driver-nvidia-gpu-component=kubelet-plugin --all-containers --prefix --tail=400
• nvidia-smi output from the host
• Kubelet logs

IMEX / ComputeDomain Debug Information (if applicable)

• Host IMEX service disabled: systemctl status nvidia-imex.service (must be masked/disabled)
• Node clique labels: kubectl get nodes -L nvidia.com/gpu.clique
• Per-GPU clique info: nvidia-smi -q | grep -E 'ClusterUUID|CliqueId'
• NVLink/NVSwitch topology: nvidia-smi topo -m
• ComputeDomain status: kubectl get computedomains.resource.nvidia.com -o yaml
• ComputeDomainClique status: kubectl get computedomaincliques.resource.nvidia.com -o yaml
• IMEX daemon pods: kubectl get pods -n <namespace> -l resource.nvidia.com/computeDomain
• IMEX daemon logs: kubectl logs -n dra-driver-nvidia-gpu -l resource.nvidia.com/computeDomain --all-containers --prefix --tail=-1
• IMEX domain status: nvidia-imex-ctl -c /etc/nvidia-imex/config.cfg -N

[shivamerla]

@aditighag I’ll take a closer look at this tomorrow. Is there a specific reason you’re testing with IMEXDaemonsWithDNSNames disabled? Do you have a use case where the workload replicas are fixed upfront and not expected to scale?

Starting with 25.12.0, this feature flag is enabled by default, and several optimizations were made around it. The main tradeoff is that workloads now need to enforce their own barriers to ensure all ranks are ready, rather than relying on CD status.

[aditighag]

Is there a specific reason you’re testing with IMEXDaemonsWithDNSNames disabled?

There seems to be some problem with the DNS resolution of these hosts that I haven't quite managed to track down yet.

ICYMI, I've provided a unit test repro here for the original issue - #1049.

[Apr 20 2026 19:05:12] [INFO] [tid 141] Attempting connection for target node 13 - compute-domain-daemon-0013:50000, attempt #557516
[Apr 20 2026 19:05:13] [INFO] [tid 141] Attempting connection for target node 15 - compute-domain-daemon-0015:50000, attempt #557488
[Apr 20 2026 19:05:13] [INFO] [tid 141] Attempting connection for target node 10 - compute-domain-daemon-0010:50000, attempt #557524
[Apr 20 2026 19:05:13] [INFO] [tid 141] Attempting connection for target node 11 - compute-domain-daemon-0011:50000, attempt #557496
[Apr 20 2026 19:05:14] [INFO] [tid 141] Attempting connection for target node 16 - compute-domain-daemon-0016:50000, attempt #557437
[Apr 20 2026 19:05:14] [INFO] [tid 141] Attempting connection for target node 4 - compute-domain-daemon-0004:50000, attempt #557518
[Apr 20 2026 19:05:15] [INFO] [tid 141] Attempting connection for target node 3 - compute-domain-daemon-0003:50000, attempt #557518
[Apr 20 2026 19:05:15] [INFO] [tid 141] Attempting connection for target node 2 - compute-domain-daemon-0002:50000, attempt #557470

[shivamerla]

Is there a specific reason you’re testing with IMEXDaemonsWithDNSNames disabled?

There seems to be some problem with the DNS resolution of these hosts that I haven't quite managed to track down yet.

ICYMI, I've provided a unit test repro here for the original issue - #1049.

[Apr 20 2026 19:05:12] [INFO] [tid 141] Attempting connection for target node 13 - compute-domain-daemon-0013:50000, attempt #557516
[Apr 20 2026 19:05:13] [INFO] [tid 141] Attempting connection for target node 15 - compute-domain-daemon-0015:50000, attempt #557488
[Apr 20 2026 19:05:13] [INFO] [tid 141] Attempting connection for target node 10 - compute-domain-daemon-0010:50000, attempt #557524
[Apr 20 2026 19:05:13] [INFO] [tid 141] Attempting connection for target node 11 - compute-domain-daemon-0011:50000, attempt #557496
[Apr 20 2026 19:05:14] [INFO] [tid 141] Attempting connection for target node 16 - compute-domain-daemon-0016:50000, attempt #557437
[Apr 20 2026 19:05:14] [INFO] [tid 141] Attempting connection for target node 4 - compute-domain-daemon-0004:50000, attempt #557518
[Apr 20 2026 19:05:15] [INFO] [tid 141] Attempting connection for target node 3 - compute-domain-daemon-0003:50000, attempt #557518
[Apr 20 2026 19:05:15] [INFO] [tid 141] Attempting connection for target node 2 - compute-domain-daemon-0002:50000, attempt #557470

@aditighag these errors are expected as we add fixed number of DNS names (18 nodes) within each IMEX domain. These errors appear if there is no corresponding pod IP mapped to it (due to a smaller domain formation for the running workload). Apart from these messages in the log, did you see domain not converging in the CD status with IMEXDaemonsWithDNSNames enabled? The reason i insist on this is, there has been significant improvement with it enabled and this is our recommended (default) approach going forward as well.

[jgehrcke]

Thanks for the report!

I want to underline what @shivamerla asked here:

did you see domain not converging in the CD status with IMEXDaemonsWithDNSNames enabled? The reason i insist on this is, there has been significant improvement with it enabled and this is our recommended (default) approach going forward as well.

That feedback would be very important.

If we have a bug in the IMEXDaemonsWithDNSNames=false code path and if the issue is not reproducible in the (default) IMEXDaemonsWithDNSNames=true mode and if you don't have a strong reason to actually use IMEXDaemonsWithDNSNames=false -- then the correct solution is to use IMEXDaemonsWithDNSNames=true.

If such strong reason exists: we have to know about it, and narrow it down as precisely as possible. We should then steer our energy towards making IMEXDaemonsWithDNSNames=true work for you.

[aditighag]

Hey!

The IMEXDaemonsWithDNSNames=true is working as expected based on the initial results. If you look the snippet I added to the "Expected Behavior" section in the original report, it's with IMEXDaemonsWithDNSNames enabled.

To clarify, I believe the bug still exists when this flag is disabled. One good property of IMEXDaemonsWithDNSNames=false is that it allows gating workload pods until all nodes in a ComputeDomain have joined the IMEX domain.
When IMEXDaemonsWithDNSNames is enabled without numNodes, the onus shifts to end users to investigate whether the IMEX domain for their workload (ComputeDomain) is ready. Do you have any recommendations for how this can be tracked better?