[Bug]: VFIO discovery advertises non-vfio GPUs and Unconfigure rebinds pre-bound GPUs

[johnahull]

Component: gpu-kubelet-plugin

Bug Description:

Four related bugs in the VFIO passthrough lifecycle that prevent GPU passthrough to VMs on multi-GPU and NVLink systems:

1. CDI spec missing /dev/vfio/vfio — GetCommonEdits only includes /dev/vfio/vfio when enableAPIDevice=true. Libvirt requires it to detect VFIO support regardless of the API device setting.

2. VFIO discovery advertises non-vfio GPUs — enumerateGpuVfioDevices treats any GPU not on the nvidia driver as a VFIO candidate, including driverless GPUs (stuck after a failed unbind). The scheduler allocates them, and prepare fails or hangs.

3. Unconfigure rebinds pre-bound GPUs — On H100 SXM5 with NVLink, Unconfigure tries to rebind vfio-pci GPUs back to nvidia, which hangs indefinitely during NVLink fabric reconfiguration. GPUs pre-bound to vfio-pci at boot (via vfio-pci.ids kernel cmdline) should stay on vfio-pci.

4. Sysfs checks fail inside containers — checkVfioPCIModuleLoaded and checkIommuEnabled check /host-root/sys/ which doesn't expose host sysfs inside containers. The VfioPciManager fails to initialize even though vfio_pci and IOMMU are working on the host.

Steps to Reproduce:

1. System with H100 SXM5 GPUs (NVLink), GPUs pre-bound to vfio-pci via vfio-pci.ids=10de:2330
2. Deploy NVIDIA DRA driver with PassthroughSupport=true
3. Create a ResourceClaim requesting a VFIO GPU
4. Various failures depending on which bug is hit

Expected Behavior:

VFIO prepare should succeed for GPUs already bound to vfio-pci. CDI spec should always include /dev/vfio/vfio. Only GPUs actually on vfio-pci should be advertised. Sysfs checks should work inside containers.

DRA Driver Version: v25.12.0
Kubernetes Version: v1.36.0
GPU Model: NVIDIA H100 SXM5 80GB HBM3
NVIDIA Driver Version: 595.58
OS / Kernel: Fedora 44, kernel 6.19.14
Container Runtime: containerd 2.2.3

Feature Gates: PassthroughSupport=true, DeviceMetadata=true

[varunrsekar]

@johnahull I want to address each of the 4 issues you mentioned in the bug:

CDI spec missing /dev/vfio/vfio — GetCommonEdits only includes /dev/vfio/vfio when enableAPIDevice=true. Libvirt requires it to detect VFIO support regardless of the API device setting.

This is expected behavior. The api device /dev/vfio/vfio provides added privileges to your workload. Hence its not included in the default CDI spec. For kubevirt/libvirt workloads that need it, you can work around it by creating your custom deviceclass for your claims:

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: vfio-kv.gpu.nvidia.com
spec:
  config:
  - opaque:
      driver: gpu.nvidia.com
      parameters:
        apiVersion: resource.nvidia.com/v1beta1
        iommu:
          enableAPIDevice: true
        kind: VfioDeviceConfig
  selectors:
  - cel:
      expression: device.driver == 'gpu.nvidia.com' && device.attributes['gpu.nvidia.com'].type
        == 'vfio'

VFIO discovery advertises non-vfio GPUs — enumerateGpuVfioDevices treats any GPU not on the nvidia driver as a VFIO candidate, including driverless GPUs (stuck after a failed unbind). The scheduler allocates them, and prepare fails or hangs.

I don't understand this one. Can you please elaborate on what the issue is, include logs and your repro steps?

Unconfigure rebinds pre-bound GPUs — On H100 SXM5 with NVLink, Unconfigure tries to rebind vfio-pci GPUs back to nvidia, which hangs indefinitely during NVLink fabric reconfiguration.

Please share logs and what exactly is happening in your env for this one.

GPUs pre-bound to vfio-pci at boot (via vfio-pci.ids kernel cmdline) should stay on vfio-pci.

This is an interesting usecase and I'd prefer this to be tracked independently as a feature request, not as a bug. Can you file a separate issue for this one?

Sysfs checks fail inside containers — checkVfioPCIModuleLoaded and checkIommuEnabled check /host-root/sys/ which doesn't expose host sysfs inside containers. The VfioPciManager fails to initialize even though vfio_pci and IOMMU are working on the host.

Again, please include logs for this one and your plugin pod spec. The only reason I can think of as this would fail is if you haven't mounted the host root in your plugin daemonset. DRA driver installation via the helm chart is the only supported way to use the plugin.

[johnahull]

@varunrsekar Thanks for the detailed feedback. I'll address each issue:

Issue 1: CDI spec missing /dev/vfio/vfio

▎ This is expected behavior. The api device /dev/vfio/vfio provides added privileges... you can work around it by creating your custom
▎ deviceclass with enableAPIDevice: true

That makes sense. I'll use the DeviceClass config approach and drop this change from the PR. Thanks for the example.

Issue 2: VFIO discovery advertises non-vfio GPUs

▎ I don't understand this one. Can you please elaborate on what the issue is, include logs and your repro steps?

System: Dell XE8640 with 2x NVIDIA A40 GPUs. One GPU is bound to nvidia, the other to vfio-pci:

$ lspci -d 10de: -k | grep -A2 '3D'
4a:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
Kernel driver in use: nvidia
61:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
Kernel driver in use: vfio-pci

The DRA driver startup log (kubectl logs -n nvidia-dra dra-driver-nvidia-gpu-kubelet-plugin-xvjtv -c gpus) shows both GPUs advertised
as VFIO devices:

I0424 23:31:40.935780 nvlib.go:589] Adding VFIO device for discovered GPU PCI device: 0000:4a:00.0
I0424 23:31:40.935876 allocatable.go:256] Adding allocatable device "gpu-vfio-0" for PCI bus ID: 0000:4a:00.0
I0424 23:31:40.935882 nvlib.go:589] Adding VFIO device for discovered GPU PCI device: 0000:61:00.0
I0424 23:31:40.935919 allocatable.go:256] Adding allocatable device "gpu-vfio-1" for PCI bus ID: 0000:61:00.0

Both gpu-vfio-0 and gpu-vfio-1 appear in the ResourceSlice. But gpu-vfio-0 (0000:4a:00.0) is bound to nvidia, not vfio-pci.
enumerateGpuVfioDevices doesn't check the actual kernel driver binding — it treats any GPU that isn't skipped by
parent.Gpu.vfioEnabled as a VFIO candidate.

If the scheduler allocates gpu-vfio-0, Configure has to unbind from nvidia and rebind to vfio-pci at runtime. On NVLink systems this
can hang during fabric reconfiguration.

The fix adds a getDriver() check before advertising — only GPUs actually bound to vfio-pci are advertised as VFIO devices.

Issue 3: Unconfigure rebinds pre-bound GPUs

▎ Please share logs and what exactly is happening in your env for this one.

Same XE8640 system. GPU 0000:61:00.0 is pre-bound to vfio-pci via kernel cmdline vfio-pci.ids=10de:2235. When a VM using this GPU is
deleted:

1. Unconfigure runs and calls bindToDriver("0000:61:00.0", "nvidia")
2. The nvidia driver starts NVLink fabric initialization on the GPU
3. This hangs for 30+ seconds and sometimes fails, leaving the GPU in a bad state where neither nvidia nor vfio-pci can claim it

The core fix tracks the driver that was bound before Configure ran (preConfigureDriver). If it was already vfio-pci, Unconfigure skips
the rebind — it restores the previous state rather than unconditionally rebinding to nvidia.

I'll file the vfio-pci.ids kernel cmdline detection as a separate feature request as you suggested.

Issue 4: Sysfs checks fail inside containers

▎ The only reason I can think of as this would fail is if you haven't mounted the host root in your plugin daemonset. DRA driver
▎ installation via the helm chart is the only supported way to use the plugin.

You're right — we were testing with a manually-deployed pod that didn't have /host-root mounted. With the helm chart deployment the
sysfs checks work. I'll drop the fallback change from the PR.

Updated PR plan:

I'll rework the PR to:

1. Drop the /dev/vfio/vfio change (use DeviceClass config instead)
2. Keep the VFIO discovery driver check (issue 2)
3. Keep the preConfigureDriver tracking in Unconfigure (issue 3), file vfio-pci.ids detection as a separate feature request
4. Drop the sysfs fallback (issue 4)

[johnahull]

@varunrsekar Thanks for the detailed feedback. Let me address each point.

Issue 1: CDI spec missing /dev/vfio/vfio

This is expected behavior. The api device /dev/vfio/vfio provides added privileges... you can work around it by creating your custom deviceclass with enableAPIDevice: true

That makes sense. I'll use the DeviceClass config approach and drop this change from the PR. Thanks for the example.

Issue 2: VFIO discovery advertises non-vfio GPUs

I don't understand this one. Can you please elaborate on what the issue is, include logs and your repro steps?

System: Dell XE8640 with 2x NVIDIA A40 GPUs. One GPU is bound to nvidia, the other to vfio-pci:

$ lspci -d 10de: -k | grep -A2 '3D'
4a:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
        Kernel driver in use: nvidia
61:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
        Kernel driver in use: vfio-pci

The DRA driver startup log (kubectl logs -n nvidia-dra dra-driver-nvidia-gpu-kubelet-plugin-xvjtv -c gpus) shows both GPUs advertised as VFIO devices:

I0424 23:31:40.935780  nvlib.go:589] Adding VFIO device for discovered GPU PCI device: 0000:4a:00.0
I0424 23:31:40.935876  allocatable.go:256] Adding allocatable device "gpu-vfio-0" for PCI bus ID: 0000:4a:00.0
I0424 23:31:40.935882  nvlib.go:589] Adding VFIO device for discovered GPU PCI device: 0000:61:00.0
I0424 23:31:40.935919  allocatable.go:256] Adding allocatable device "gpu-vfio-1" for PCI bus ID: 0000:61:00.0

Both gpu-vfio-0 and gpu-vfio-1 appear in the ResourceSlice. But gpu-vfio-0 (0000:4a:00.0) is bound to nvidia, not vfio-pci. enumerateGpuVfioDevices doesn't check the actual kernel driver binding — it treats any GPU that isn't skipped by parent.Gpu.vfioEnabled as a VFIO candidate.

If the scheduler allocates gpu-vfio-0, Configure has to unbind from nvidia and rebind to vfio-pci at runtime. On NVLink systems this can hang during fabric reconfiguration.

The fix adds a getDriver() check before advertising — only GPUs actually bound to vfio-pci are advertised as VFIO devices.

Issue 3: Unconfigure rebinds pre-bound GPUs

Please share logs and what exactly is happening in your env for this one.

Same XE8640 system. GPU 0000:61:00.0 is pre-bound to vfio-pci via kernel cmdline vfio-pci.ids=10de:2235. When a VM using this GPU is deleted:

1. Unconfigure runs and calls bindToDriver("0000:61:00.0", "nvidia")
2. The nvidia driver starts NVLink fabric initialization on the GPU
3. This hangs for 30+ seconds and sometimes fails, leaving the GPU in a bad state where neither nvidia nor vfio-pci can claim it

The core fix tracks the driver that was bound before Configure ran (preConfigureDriver). If it was already vfio-pci, Unconfigure skips the rebind — it restores the previous state rather than unconditionally rebinding to nvidia.

I've filed the vfio-pci.ids kernel cmdline detection as a separate feature request: #1099

Issue 4: Sysfs checks fail inside containers

The only reason I can think of as this would fail is if you haven't mounted the host root in your plugin daemonset. DRA driver installation via the helm chart is the only supported way to use the plugin.

You're right — we were testing with a manually-deployed pod that didn't have /host-root mounted. With the helm chart deployment the sysfs checks work. I'll drop the fallback change from the PR.

Updated PR:

I've reworked #1090 to only include the two bug fixes (issues 2 and 3), dropping the changes for issues 1 and 4.

[johnahull]

@varunrsekar Thanks for the detailed feedback. Let me address each point.

Issue 1: CDI spec missing /dev/vfio/vfio

This is expected behavior... you can work around it by creating your custom deviceclass with enableAPIDevice: true

That makes sense. I'll use the DeviceClass config approach. Dropped from the PR.

Issue 2: VFIO discovery advertises non-vfio GPUs

I don't understand this one. Can you please elaborate on what the issue is, include logs and your repro steps?

System: Dell XE8640 with 2x NVIDIA A40 GPUs. One on nvidia, the other on vfio-pci:

$ lspci -d 10de: -k | grep -A2 '3D'
4a:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
        Kernel driver in use: nvidia
61:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
        Kernel driver in use: vfio-pci

DRA driver startup log (kubectl logs -n nvidia-dra dra-driver-nvidia-gpu-kubelet-plugin-xvjtv -c gpus):

I0424 23:31:40.935780  nvlib.go:589] Adding VFIO device for discovered GPU PCI device: 0000:4a:00.0
I0424 23:31:40.935876  allocatable.go:256] Adding allocatable device "gpu-vfio-0" for PCI bus ID: 0000:4a:00.0
I0424 23:31:40.935882  nvlib.go:589] Adding VFIO device for discovered GPU PCI device: 0000:61:00.0
I0424 23:31:40.935919  allocatable.go:256] Adding allocatable device "gpu-vfio-1" for PCI bus ID: 0000:61:00.0

gpu-vfio-0 (0000:4a:00.0) is bound to nvidia, not vfio-pci, but is advertised as a VFIO device. enumerateGpuVfioDevices doesn't check the actual driver binding. If the scheduler allocates it, Configure has to unbind from nvidia and rebind to vfio-pci at runtime, which hangs on NVLink systems.

Fix: add a getDriver() check — only GPUs actually bound to vfio-pci are advertised.

Issue 3: Unconfigure rebinds pre-bound GPUs

Please share logs and what exactly is happening in your env for this one.

Same system. GPU 0000:61:00.0 is pre-bound to vfio-pci via kernel cmdline vfio-pci.ids=10de:2235. When a VM is deleted, Unconfigure calls bindToDriver("0000:61:00.0", "nvidia"). On NVLink systems this triggers fabric reconfiguration that hangs 30+ seconds and can leave the GPU unusable.

Fix: track the driver bound before Configure ran (preConfigureDriver). If it was already vfio-pci, skip the rebind.

Filed the vfio-pci.ids kernel cmdline detection as a separate feature request: #1099

Issue 4: Sysfs checks fail inside containers

The only reason I can think of as this would fail is if you haven't mounted the host root in your plugin daemonset.

You're right — we were testing with a manually-deployed pod without /host-root. Dropped from the PR.

Updated PR: Reworked #1090 to only include issues 2 and 3.

[shivamerla]

/assign @varunrsekar