CSI won't handle resize of pvc

[markussiebert]

Bug Report

Trying to resize a pvc, but csi won't handle:

Name:          some-postgres-1                                                                                                                                                                                                                                                                                                                                                                                
Namespace:     app-some                                                                                                                                                                                                                                                                                                                                                                                       
StorageClass:  proxmox-data-xfs                                                                                                                                                                                                                                                                                                                                                                                 
Status:        Bound                                                                                                                                                                                                                                                                                                                                                                                            
Volume:        pvc-811e63d1-45e3-4987-a14c-72f359960887                                                                                                                                                                                                                                                                                                                                                         
Labels:        cnpg.io/cluster=some-postgres                                                                                                                                                                                                                                                                                                                                                                  
               cnpg.io/instanceName=some-postgres-1                                                                                                                                                                                                                                                                                                                                                           
               cnpg.io/instanceRole=primary                                                                                                                                                                                                                                                                                                                                                                     
               cnpg.io/pvcRole=PG_DATA                                                                                                                                                                                                                                                                                                                                                                          
               role=primary                                                                                                                                                                                                                                                                                                                                                                                     
Annotations:   cnpg.io/nodeSerial: 1                                                                                                                                                                                                                                                                                                                                                                            
               cnpg.io/operatorVersion: 1.26.0                                                                                                                                                                                                                                                                                                                                                                  
               cnpg.io/pvcStatus: ready                                                                                                                                                                                                                                                                                                                                                                         
               pv.kubernetes.io/bind-completed: yes                                                                                                                                                                                                                                                                                                                                                             
               pv.kubernetes.io/bound-by-controller: yes                                                                                                                                                                                                                                                                                                                                                        
               volume.beta.kubernetes.io/storage-provisioner: csi.proxmox.sinextra.dev                                                                                                                                                                                                                                                                                                                          
               volume.kubernetes.io/selected-node: talos-production                                                                                                                                                                                                                                                                                                                                             
               volume.kubernetes.io/storage-provisioner: csi.proxmox.sinextra.dev                                                                                                                                                                                                                                                                                                                               
Finalizers:    [kubernetes.io/pvc-protection]                                                                                                                                                                                                                                                                                                                                                                   
Capacity:      16Gi                                                                                                                                                                                                                                                                                                                                                                                             
Access Modes:  RWO                                                                                                                                                                                                                                                                                                                                                                                              
VolumeMode:    Filesystem                                                                                                                                                                                                                                                                                                                                                                                       
Used By:       some-postgres-1                                                                                                                                                                                                                                                                                                                                                                                
Events:                                                                                                                                                                                                                                                                                                                                                                                                         
  Type     Reason             Age   From           Message                                                                                                                                                                                                                                                                                                                                                      
  ----     ------             ----  ----           -------                                                                                                                                                                                                                                                                                                                                                      
  Warning  ExternalExpanding  59m   volume_expand  waiting for an external controller to expand this PVC  

Description

The volume won't resize ...

Logs

I can see the volume in the logs in a line sarting with:

csi-provisioner I0529 17:05:46.698610       1 controller.go:1171] "handleProtectionFinalizer"

No other mentions of that pvc ... and no resize

Environment

• Plugin version:
• Kubernetes version:

Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.32.3

• CSI capasity:

proxmox-data-xfs   41267203408   map[topology.kubernetes.io/region:pve topology.kubernetes.io/zone:pve]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request

[sergelogvinov]

Hello,

It would be best to start by collecting the logs. The resize process involves two containers within the proxmox-csi-plugin-controller pod: csi-resizer and proxmox-csi-plugin-controller. Please check the logs for both containers. If the logs don’t provide enough detail, consider increasing the verbosity level by adding -v=5.

Thank you!

[tamcore]

Can confirm. Going through my helm releases and downgrading 1 by 1 fixed it, once i made it to 0.2.14, which is version v0.8.2 of the controller.

Will try to upgrade and collect logs later.

[tamcore]

I've attached log files for both containers. But they don't seem to show anything useful.

The first attempt to resize test-pvc somehow succeeded. Continued attempts, just didn't.

csi-resizer.log
proxmox-csi-controller.log

[sergelogvinov]

Hello,
I found the logs

proxmox-csi-controller-7fcdb8fcd5-66f9l proxmox-csi-controller I0601 17:31:35.726473       1 controller.go:757] "ControllerExpandVolume: volume expanded" cluster="home" volumeID="home/pve/tank/vm-9999-pvc-c10754fc-eb36-4c2e-8ef9-bd83914eb247" vmID=1006 size=2147483648

Can you check if the PV pv=vm-9999-pvc-c10754fc-eb36-4c2e-8ef9-bd83914eb247 had its size changed?

[tamcore]

Unfortunately not. No change on the PV, or even the status of the PVC.

Currently it also looks like

---
kind: PersistentVolumeClaim
spec:
  resources:
    requests:
      storage: 3Gi
status:
  allocatedResourceStatuses:
    storage: NodeResizePending
  allocatedResources:
    storage: 2Gi
  capacity:
    storage: 2Gi
---
kind: PersistentVolume
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi

and the only Event in the namespace is also just

80s         Normal    ExternalExpanding              persistentvolumeclaim/test-pvc   waiting for an external controller to expand this PVC

without anything else in the logs.

[sergelogvinov]

Please try to collect the logs about 2 minutes after the resize request. it doesn’t happen instantly...

[tamcore]

Unfortunately it doesn't make a difference. Nothing changed in the 20 minutes since my previous comment. If you look at the logs I posted yesterday, you'll find that those were also from a 10+ minute timeframe.

And in a cluster that is not loaded, it does indeed happen almost instantly.

But, I think i can consistently reproduce the issue between version 0.3.3 and 0.3.4 of the chart. If i downgrade to 0.3.3, everything works and resizes basically instantly. If i upgrade to 0.3.4, it ceases to work.

[tamcore]

Using the latest version 0.3.7 of the chart (with v0.11.0 of the controller) with controller.resizer.image.tag=v1.12.0, everything works. If i upgrade it to v1.13.1 again, resizing is broken again.

@markussiebert can you try to verify this on your end, by setting controller.resizer.image.tag=v1.12.0 for your installation?

[markussiebert]

Yes, instantly volume appears in the logs and volume was resized.

[sergelogvinov]

Ok, the issue is that the resizer sidecar (v1.13.1) is incompatible with the plugin version (v0.11.0) (i hope)

I'm going to make a new release soon - it will probably resolve the issue.

[sergelogvinov]

I've tested the latest release (0.12.0), and PVC resizing works as expected.

[tamcore]

Can't confirm. Upgraded to 0.3.9 of the chart with proxmox-csi-controller:v0.12.0 with csi-resizer:v1.13.2 and tried to resize a volume - nothing happened. Downgraded the csi-resizer again to v1.12.0 and it worked again immediatley.

[sergelogvinov]

I've tested it in my cluster — after a short delay, the disk is resized as expected. It's possible that downgrading the version triggered a restart of the csi-resizer pod, which in turn sent the resize notification to the controller.

[tamcore]

Nothing happening here. Not even after waiting for 10 minutes. Killing the proxmox-csi-controller-* pod to force a restart of the csi-resizer container also doesn't change anything.

This is on K3s 1.32.2 and K3s 1.32.5 btw. Upgraded just in case.

Downgrading to v0.12.0 makes it work again immediatley.

[tamcore]

With proxmox-csi-controller:v0.12.1 and csi-resizer:v1.13.2- as installed by version 0.3.10 of the chart - it seems to work now.