Skip to content

cinder csi v1.28.1 multi attacg volume error #3001

@Utmishra12345

Description

@Utmishra12345

"/var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0924 07:27:21.849119 1 mount_linux.go:360] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount
I0924 07:27:21.850774 1 mount_linux.go:828] ignoring 'not mounted' error for /var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount
I0924 07:27:21.850790 1 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount"
E0924 07:27:21.850824 1 utils.go:95] [ID:3101502] GRPC error: rpc error: code = Internal desc = Unmount of targetPath /var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount failed with error remove /var/lib/kubelet/plugins/kubernetes.io/csi/cinder.csi.openstack.org/3ee57b7499fe262d4834be36cf03e5b2e76555f53bfb94401155ab507dbfca06/globalmount: directory not empty

facing the above logs from cinder csi while scaling up and scaling down the pods
want to know if it is a bug in the cinder csi plugin on version 1.28.1 ,it will be really helpfull or it is a cinder csi issue or some ither issue - below is my detailed anallysis

Kubelet’s role
Kubelet orchestrates volume lifecycle, but the actual mount/unmount logic for Cinder is delegated to the CSI plugin (cinder-csi-nodeplugin and controller). When kubelet shows:
• ignoring 'not mounted' error
• GRPC error: Unmount ... failed with error remove .../globalmount: directory not empty
it means kubelet asked the CSI driver to clean up. CSI did not finish (directory not empty → volume still busy).
• What kubelet did correctly
o Sent preStop to the container.
o Waited for terminationGracePeriod.
o Tried to unmount the volume via CSI.
o Logged the CSI failure.
o When grace period expired, killed the container (exit 137).
• Where the failure is
o The Cinder CSI nodeplugin should have cleaned up the globalmount directory.
o OpenStack should have marked the volume detached. Instead, it stayed in-use.
o Leftover files (config db logs) prove CSI unpublish didn’t fully succeed.
• Conclusion
Kubelet is just the messenger here — it times out because CSI cleanup stalls. The root cause is in the CSI/OpenStack layer (either driver bug, infra race condition, or OpenStack not completing detach).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions