-
Couldn't load subscription status.
- Fork 648
Description
"/var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0924 07:27:21.849119 1 mount_linux.go:360] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount
I0924 07:27:21.850774 1 mount_linux.go:828] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount
I0924 07:27:21.850790 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount"
E0924 07:27:21.850824 1 utils.go:95] [ID:3101502] GRPC error: rpc error: code = Internal desc = Unmount of targetPath /var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount failed with error remove /var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount: directory not empty
facing the above logs from cinder csi while scaling up and scaling down the pods
want to know if it is a bug in the cinder csi plugin on version 1.28.1 ,it will be really helpfull or it is a cinder csi issue or some ither issue - below is my detailed anallysis
Kubelet’s role
Kubelet orchestrates volume lifecycle, but the actual mount/unmount logic for Cinder is delegated to the CSI plugin (cinder-csi-nodeplugin and controller). When kubelet shows:
• ignoring 'not mounted' error
• GRPC error: Unmount ... failed with error remove .../globalmount: directory not empty
it means kubelet asked the CSI driver to clean up. CSI did not finish (directory not empty → volume still busy).
• What kubelet did correctly
o Sent preStop to the container.
o Waited for terminationGracePeriod.
o Tried to unmount the volume via CSI.
o Logged the CSI failure.
o When grace period expired, killed the container (exit 137).
• Where the failure is
o The Cinder CSI nodeplugin should have cleaned up the globalmount directory.
o OpenStack should have marked the volume detached. Instead, it stayed in-use.
o Leftover files (config db logs) prove CSI unpublish didn’t fully succeed.
• Conclusion
Kubelet is just the messenger here — it times out because CSI cleanup stalls. The root cause is in the CSI/OpenStack layer (either driver bug, infra race condition, or OpenStack not completing detach).