Ceph-csi-rbd on k3s gives timeout on reboot

[Ramshield]

Describe the bug

I'm running k3s with ceph-csi-rbd. When I reboot the Virtual Machine on my Proxmox cluster I get "libceph: connect (1)10.255.255.2:6789 error -101" and "mon1 (1)10.255.255.5:6789 connect error" spamming errors on the console for about 5 minutes, before it acutally reboots.

Environment details

• Image/version of Ceph CSI driver : Which container exactly?
• Helm chart version : 3.15.0
• Kernel version :
• Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
  krbd or rbd-nbd) :
• Kubernetes cluster version : v1.33.4+k3s1
• Ceph cluster version : 19.2.3

Steps to reproduce

Steps to reproduce the behavior:

3 node cluster with 1 control plane, helm chart values:

csiConfig:
  - clusterID: <HIDDEN>
    monitors:
      - '10.255.255.5:6789'
      - '10.255.255.2:6789'
      - '10.255.255.3:6789'
provisioner:
  replicaCount: 1
secret:
  create: false  # Using an external secret from hashicorp vault
selinuxMount: false
storageClass:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  clusterID: <HIDDEN>
  create: true
  name: csi-cephrbd-sc
  pool: k8s_rbd_pool

Actual results

See screenshot above.

Expected behavior

A normal, quick reboot.

Logs

Ceph monitoring (of 10.255.255.5) reports normal logging status messages;
2025-10-20T09:46:53.229034+0200 mgr.kvm-03 (mgr.120307575) 1956 : cluster [DBG] pgmap v1969: 417 pgs: 417 active+clean; 4.0 TiB data, 12 TiB used, 15 TiB / 27 TiB avail; 445 KiB/s rd, 1.0 MiB/s wr, 57 op/s

The provisioner doesn't log any errors (except VolumeSnapshot errors, as I didn't install the VolumeSnapshotClass CRD's or anything).

Additional context

Add any other context about the problem here.

For example:

Any existing bug report which describe about the similar issue/behavior

[Madhu-1]

@Ramshield is this the kubernetes node where you are seeing this error? This is mostly network issue, once volume creation and mounting is done csi is out of the picture and if you are using kernel client there should not be any problem. we add the netdev from very long time to make sure the network is disconnected at last. I also assume that /run/mount/utab is shared with host and the nodeplugin, please see that if there are any manifest issues for it. linking #756 to provide more details to you.

[Ramshield]

@Madhu-1 It is indeed the Kubernetes node, and it happens on both the control-plane and the worker. I'm not sure what you mean with network issue, as this is when rebooting a Virtual Machine.
I did not specifically shared /run/mount/utab. If that's in the Helm chart, then yes.

But thank you for your quick response, I will ready through the issue you provided.

[Ramshield]

@Madhu-1 I just checked and confirmed, /run/mount and the file utab is present on all the csi-rbdplugin pods.

The K3s hosts are Debian 13, kernel:

$ uname -a
Linux k3s-01 6.12.48+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.48-1 (2025-09-20) x86_64 GNU/Linux

Is there anything else I can do to fix this?

[Madhu-1]

@Madhu-1 I just checked and confirmed, /run/mount and the file utab is present on all the csi-rbdplugin pods.

The K3s hosts are Debian 13, kernel:

$ uname -a
Linux k3s-01 6.12.48+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.48-1 (2025-09-20) x86_64 GNU/Linux

Is there anything else I can do to fix this?

Did you verify the file also exists on the node with same contents?

[Ramshield]

I just verified the file exists on the Kubernetes host with exactly the same contents.