Bug Report: Nested mountpoints causes failure when using filesystem mode local PVs

[nitishSubramanian]

What happened

When using a StorageClass with the local-static-provisioner and volumeMode: Filesystem, the provisioner creates mount points under:

/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~local-volume/<volume-name>

If the underlying filesystem already contains older or nested mount points:

• Existing (older) mount points become inaccessible from the Pod.
• Newly created nested mount points cause the kubelet to fail during unmount or detach operations, preventing the Pod from terminating cleanly.

What you expected to happen

Local PersistentVolumes provisioned with volumeMode: Filesystem should:

• Use isolated or predictable mount locations that avoid nesting mount points. (by using symlink)
• Properly unmount and clean up all filesystem mount points when Pods terminate.

How to reproduce it

1. Deploy a Pod that uses a Local PV provisioned via a StorageClass configured for the local-static-provisioner.

2. Ensure the PV path includes a mount point created using --bind.

3. Confirm the mount point is accessible from within the Pod.

4. On the node, create a new nested bind mount under the same path.

5. Delete the Pod.

6. Observe that the unmount operation fails. The kubelet logs show errors similar to:

   umount: /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~local-volume/<volume-name>: target is busy
   

Anything else we need to know

This issue results in stale mounts under /var/lib/kubelet/pods, blocking pod cleanup and re-scheduling.

Environment

• Kubernetes version: v1.29
• OS: AL2023
• Kernel: 6.12.40-64.114.amzn2023.x86_64