Controler failed to make subdirectory: mkdir /tmp/pvc-xxxx: read-only file system

[seb-835]

What happened:

The CSI-NFS controler refused to create the mount directory in container :

E1122 16:36:41.400529       1 utils.go:91] GRPC error: rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-68f3e65a-b286-4d97-94d3-534e260f0568/pvc-68f3e65a-b286-4d97-94d3-534e260f0568: read-only file system

But when i attach to the container

kubectl exec -it csi-nfs-controller-57594f8fd-pr9d7 -n kube-system -c nfs -- /bin/sh

and make a mkdir /tmp/test, it works , and i can mount the sharing.

I am sure , i forget something, but i can´t put the finger on, could you help please ?

Environment:

• CSI Driver version: 3.0.0

• Kubernetes version : 1.21.4

• OS : CENTOS 7.9

• Kernel : 5.13.0-21-generic

• sestatus

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

CSI installed using HELM-3 :

helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --set controller.runOnMaster=true --set controller.replicas=3

[andyzhangx]

does /tmp/pvc-68f3e65a-b286-4d97-94d3-534e260f0568/pvc-68f3e65a-b286-4d97-94d3-534e260f0568 readonly, can you make a directory under that dir?

[seb-835]

The directory /tmp/pvc-68f3e65a-b286-4d97-94d3-534e260f0568/ does not exists in tmp that's why i think it fails to create the second subdirectory : pvc-68f3e65a-b286-4d97-94d3-534e260f0568 in (with the same name ?)

Is it ok for you that controller build this : /tmp/pvc-68f3e65a-b286-4d97-94d3-534e260f0568/pvc-68f3e65a-b286-4d97-94d3-534e260f0568 ?

mkdir -p /tmp/pvc-68f3e65a-b286-4d97-94d3-534e260f0568/pvc-68f3e65a-b286-4d97-94d3-534e260f0568 is OK

[andyzhangx]

the error was caused by: this driver does not have permission to mkdir on the nfs server

[seb-835]

oh damn, i feel so stupid !!! i close the ticket , thanks for the time spent on this stupid error.

[akagr]

@andyzhangx I'm seeing this error and I don't understand. Why do we need permissions to create directories on NFS server? We already have some directories under which we can create files. I'm getting this exact same error. Bascially, the server is exporting the share as / and it has a few directories under it. For example /dir1, /dir2 etc. We can mount / and create files under dir1 and dir2 manually.

But csi driver is trying to create /pvc-*** directory, which is not allowed. Something I can configure it shouldn't do?

[andyzhangx]

@akagr you could specify subDir with existing sub dir name, then this driver won't create a new subdir, if sub directory does not exist, this driver would create a new one

[akagr]

@andyzhangx Thanks! Yeah I found the relevant bits in documentation. Had to specify the subDir. For anyone who stumbles on this, here's what my storage class looks like:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-nfs-sc
provisioner: nfs.csi.k8s.io
parameters:
  server: 192.168.1.56
  share: /
  subDir: /
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4