[Bug]: ResourceSlice published with no devices on GKE

[kasia-kujawa]

Component

gpu-kubelet-plugin

Bug Description

On GKE (COS image), I occasionally see a ResourceSlice published with an empty spec.devices:

metadata:
  creationTimestamp: "2026-04-03T11:14:29Z"
  generateName: gke-pool-94048ba1-gpu.nvidia.com-
  generation: 1
  name: gke-pool-94048ba1-gpu.nssw75
  ownerReferences:
  - apiVersion: v1
    controller: true
    kind: Node
    name: gke-pool-94048ba1
    uid: 8dece9be-6d55-4886-868a-b32012ad1cec
  resourceVersion: "1775214869927359005"
  uid: a4051477-ef9b-41ab-8151-55196e8d6315
spec:
  driver: gpu.nvidia.com
  nodeName: gke-pool-94048ba1
  pool:
    generation: 1
    name: gke-pool-94048ba1
    resourceSliceCount: 1

No spec.devices field at all. Restarting the nvidia-dra-driver-gpu-kubelet-plugin always fixes it, after restart the ResourceSlice contains the expected GPU devices.

The issue is not easy to reproduce, I captured it once manually, but it also happens from time to time in automated tests.

On GKE COS, the GPU driver is installed by a DaemonSet and it seems that it causes that GPU is not fully initialized when nvidia-dra-driver-gpu-kubelet-plugin starts.

The issue has never appeared on AKS or EKS, which I test in the same way. On AKS and EKS, GPU drivers are installed as part of node preparation, so the node appears with GPU drivers already installed.

Steps to Reproduce

The issue is not easy to reproduce as it happens only sometimes.

1. Install NVIDIA DRA driver on a GKE cluster with COS image (nvidiaDriverRoot: /home/kubernetes/bin/nvidia/)
2. Create a new node
3. Check the ResourceSlice

Expected Behavior

ResourceSlice contains GPU devices in spec.devices.

DRA Driver Version

v25.12.0

Kubernetes Version

1.34.4-gke.1193000

GPU Model

Tesla T4

NVIDIA Driver Version

580.105.08

OS / Kernel

COS, 6.12.55+

Container Runtime

No response

Feature Gates (non-default settings)

No response

Helm Values (non-default)

nvidiaDriverRoot: /home/kubernetes/bin/nvidia/

Relevant Logs

Debug Information Attached

• kubectl get pods -n nvidia-dra-driver-gpu -o wide
• kubectl get resourceclaims -n <namespace>
• kubectl get resourceslices.resource.k8s.io
• kubectl describe pod <pod> or kubectl events --for pod/<pod>
• Kubelet plugin logs: kubectl logs -n nvidia-dra-driver-gpu -l nvidia-dra-driver-gpu-component=kubelet-plugin --all-containers --prefix --tail=400
• nvidia-smi output from the host
• Kubelet logs

IMEX / ComputeDomain Debug Information (if applicable)

• Host IMEX service disabled: systemctl status nvidia-imex.service (must be masked/disabled)
• Node clique labels: kubectl get nodes -L nvidia.com/gpu.clique
• Per-GPU clique info: nvidia-smi -q | grep -E 'ClusterUUID|CliqueId'
• NVLink/NVSwitch topology: nvidia-smi topo -m
• ComputeDomain status: kubectl get computedomains.resource.nvidia.com -o yaml
• ComputeDomainClique status: kubectl get computedomaincliques.resource.nvidia.com -o yaml
• IMEX daemon pods: kubectl get pods -n <namespace> -l resource.nvidia.com/computeDomain
• IMEX daemon logs: kubectl logs -n nvidia-dra-driver-gpu -l resource.nvidia.com/computeDomain --all-containers --prefix --tail=-1
• IMEX domain status: nvidia-imex-ctl -c /etc/nvidia-imex/config.cfg -N

[shivamerla]

In the pre-start script recently we have changed to use nvidia-smi --version (for VFIO support) instead of nvidia-smi (which causes full GPU initialization). Wondering if that is causing the race condition here. cc @varunrsekar @kasia-kujawa the suggested fix looks good as well.

[varunrsekar]

@shivamerla the bug is filed for v25.12 so I don’t think it’s related to the pre-start changes.

@kasia-kujawa can you please provide the gpu plugin logs? And also the output of nvidia-smi on your node.

[kasia-kujawa]

@varunrsekar please see here logs from the situation when the issue happened
https://gist.github.com/kasia-kujawa/1082b48357a0ae80d663f12ee665e34c (logs come from the version with retries - #1009 and 1 retry was enough)
so the issue happens for the last released version and also for the version built with the latest code

init container logs shows the output of nvidia-smi - output was returned, exit code was 0

and in logs from gpus container please pay attention to this part:

2026-04-10 08:11:31.516	
        I0410 08:11:16.614833       1 nvlib.go:197] Traverse GPU devices
2026-04-10 08:11:31.516	
        I0410 08:11:16.779628       1 device_state.go:96] No GPU devices found yet (driver may still be initializing), retrying in 5s...
2026-04-10 08:11:31.516	
        I0410 08:11:21.780288       1 nvlib.go:197] Traverse GPU devices
2026-04-10 08:11:31.516	
        I0410 08:11:23.111832       1 nvlib.go:278] Adding device gpu-0 to allocatable device

[varunrsekar]

Thanks @kasia-kujawa I guess I was trying to understand the exact scenario where this happens. This is my best guess:

1. Use containerized nvidia driver installation
2. Scale in a new node into the cluster
3. Both driver container and DRA plugin container starts at the same time
4. While the nvidia driver installation is happening, the DRA plugin init-container crashloops
5. Once the nvidia driver installation is complete but before the GPUs are bound to it
   1. the DRA plugin init-container completes
   2. the DRA plugin container starts and comes up empty during device discovery
6. All the GPUs are bound to the nvidia driver
   1. Retry logic in the DRA plugin eventually discovers the GPUs

Can you verify if this is what's happening in your setup? If not, can you share what the scenario is?

[kasia-kujawa]

@varunrsekar you described the scenario which happens, I'll only add some small details

While the nvidia driver installation is happening, the DRA plugin init-container crashloops

No, init-container waits for drivers to be installed without restarts, if I see correctly this loop causes the it waits

dra-driver-nvidia-gpu/hack/kubelet-plugin-prestart.sh

Lines 165 to 170 in 5d2b054

	while true
	do
	validate_and_exit_on_success
	sleep ${_WAIT_S}
	_ATTEMPT=$((_ATTEMPT+1))
	done

in attached logs there are attempts logged (checking the script not all attempts are printed):

one with failure, one with success:

[init] init-container logs:
2026-04-10 08:11:31.468	
        create symlink: /driver-root -> /driver-root-parent/nvidia
2026-04-10 08:11:31.468	
        2026-04-10T08:11:01Z  /driver-root (/home/kubernetes/bin/nvidia/ on host): nvidia-smi: '/driver-root/bin/nvidia-smi', libnvidia-ml.so.1: not found, current contents: [].
2026-04-10 08:11:31.468	
        
2026-04-10 08:11:31.468	
        Check failed. Has the NVIDIA GPU driver been set up? It is expected to be installed under NVIDIA_DRIVER_ROOT (currently set to '/home/kubernetes/bin/nvidia/') in the host filesystem. If that path appears to be unexpected: review the DRA driver's 'nvidiaDriverRoot' Helm chart variable. Otherwise, review if the GPU driver has actually been installed under that path.
2026-04-10 08:11:31.468	
        Hint: Directory /home/kubernetes/bin/nvidia/ is not empty but at least one of the binaries wasn't found.
2026-04-10 08:11:31.468	
        
2026-04-10 08:11:31.468	
        2026-04-10T08:11:11Z  /driver-root (/home/kubernetes/bin/nvidia/ on host): nvidia-smi: '/driver-root/bin/nvidia-smi', libnvidia-ml.so.1: '/driver-root/lib64/libnvidia-ml.so.1', current contents: [].
2026-04-10 08:11:31.468	
        invoke: env -i LD_PRELOAD=/driver-root/lib64/libnvidia-ml.so.1 /driver-root/bin/nvidia-smi --version
2026-04-10 08:11:31.468	
        NVIDIA-SMI version  : 580.105.08
2026-04-10 08:11:31.468	
        NVML version        : 580.105
2026-04-10 08:11:31.468	
        DRIVER version      : 580.105.08
2026-04-10 08:11:31.468	
        CUDA Version        : N/A
2026-04-10 08:11:31.468	
        nvidia-smi returned with code 0: success, leave
2026-04-10 08:11:31.491

Once the nvidia driver installation is complete but before the GPUs are bound to it

It seems that something is not fully initialized.
I did some testing investigating the issue.
Earlier just nvidia-smi command was used (the issue happens also for earlier version) and nvidia-smi will report non-zero exist code if devices are not available

# nvidia-smi
No devices were found
# echo $?
6

so it looks like devices were available but not fully initialized, something was not initialized for NVML lib?

In the K8s cluster managed with cluster autoscaler the GPU node with empty ResourceSlice is useless, scheduler can't allocate GPU workloads on it, and people pay for an useable GPU node 💸 and when everything is automated 🤖 it is not good to ask people for restarting the pod 😅

So if you have another idea for fixing the issue I'm happy to test it ;)