Maintain CD daemon info in a new per-CD, per-clique ComputeDomainClique object

[klueska]

This PR introduces a new ComputeDomainClique CRD to store daemon information in separate per-clique objects rather than directly in ComputeDomain.Status.Nodes. This architectural change improves scalability by sharding daemon state across multiple objects and enables better ownership tracking through Kubernetes owner references. All changes are hidden behind the ComputeDomainCliques feature gate (default: disabled).

[jgehrcke]

Connecting dots: we motivate this architectural change in #829 (and show initial measurement results).

[jgehrcke]

LVGTM

(for anyone following this: this PR was now re-worked by Kevin into a state where the CD object contents appear just as before this patch)

• I tested this second generation of this PR manually with the 'performance/scaling testing' (analogue to what's shown in #829). Still has the same scaling characteristics as the first generation.
• We discussed the 2nd-gen approach at length, and feel good about it. We agreed that we want to iterate on details from here in separate patches. I skimmed over the code one more time and it looks good.
• Ran the complete test suite locally; patched another log mismatch (used the opportunity to also improve the log message while having to change things anyway).

Once CI passes, let's merge.

edit: @klueska maybe fix the failing DCO check and then we can merge finally :)

[jgehrcke]

✔️

Full test suite run output:

test_basics.bats
 ✓ test VERSION_W_COMMIT, VERSION_GHCR_CHART, VERSION [214]
 ✓ confirm no kubelet plugin pods running [182]
 ✓ GPU Operator installed [185]
 ✓ helm-install deployments/helm/nvidia-dra-driver-gpu/25.12.0-dev [12451]
 ✓ helm list: validate output [244]
 ✓ get crd computedomains.resource.nvidia.com [168]
 ✓ wait for plugin & controller pods READY [765]
 ✓ validate CD controller container image spec [174]
 ✓ SIGUSR2 handler: GPU plugin, CD plugin [752]
test_gpu_basic.bats
 ✓ 1 pod(s), 1 full GPU [4441]
 ✓ 2 pod(s), 1 full GPU each [5078]
 ✓ 2 pod(s), 1 full GPU (shared, 1 RC) [4996]
 ✓ 1 pod(s), 2 cntrs, 1 full GPU (shared, 1 RCT) [4645]
test_gpu_mig.bats
 ✓ static MIG: allocate (1 cnt) [16977]
 ✓ static MIG: mutual exclusivity with physical GPU [29074]
test_cd_imex_chan_inject.bats
 ✓ IMEX channel injection (single) [14664]
 ✓ IMEX channel injection (all) [8989]
test_cd_mnnvl_workload.bats
 ✓ nickelpie (NCCL send/recv/broadcast, 2 pods, 2 nodes, small payload) [11255]
 ✓ nvbandwidth (2 nodes, 2 GPUs each) [13148]
test_cd_misc.bats
 ✓ CD daemon shutdown: confirm CD status cleanup [8557]
 ✓ reject unknown field in opaque cfg in CD chan ResourceClaim [17830]
 ✓ self-initiated unprepare of stale RCs in PrepareStarted [39764]
 ✓ global CD status [9008]
 ✓ IMEX channel injection (featureGates.ComputeDomainClique=true) [22254]
test_cd_logging.bats
 ✓ CD controller/plugin: startup config / detail in logs on level 0 [13742]
 ✓ CD controller: test log verbosity levels [116528]
 ✓ CD daemon: test log verbosity levels [46115]
test_cd_failover.bats
 ✓ CD failover nvb2: force-delete worker pod 0 [49041]
 ✓ CD failover nvb2: force-delete all IMEX daemons [52070]
 ✓ CD failover nvb2: regular-delete worker pod 1 [51127]
test_cd_updowngrade.bats
 ✓ downgrade: current-dev -> last-stable [24267]
 ✓ upgrade: wipe-state, install-last-stable, upgrade-to-current-dev [39086]
test_gpu_stress.bats
 ✓ Stress: shared ResourceClaim across 15 pods x 5 loops [155523]

33 tests, 0 failures in 824 seconds

BATS_RUN_TMPDIR: /tmp/k8s-dra-driver-gpu-tests-out-jgehrcke/bats-tests-1769539733/bats-run-F5IzGR
make[1]: Leaving directory '/home/jgehrcke/dev/k8s-dra-driver-gpu'

[jgehrcke]

It seems like this needs another approval, or github is slow.

retry

[jgehrcke]

another attempt to approve (and for github to understand that)