CSI controller not able to attach disks to VMs in a different resource group

[sarandia]

What happened:
Our setup is:
azuredisk-csi-controller's cloud config points to the resource group A.
A storage class S does not have the "resourceGroup" field specified and defaults to the resource group A. We see disks created in resource group A.
However, we have VMs in resource group B, and they are not able to attach the disks with the following errors:

  Warning  FailedAttachVolume  14s (x6 over 33s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-xxxxxxxx" : rpc error: code = NotFound desc = failed to get azure instance id for node "xxxxxx" 
(instance not found)

We then found this in the CSI controller logs:

I0528 05:48:10.657321       1 csi_handler.go:243] "Error processing" VolumeAttachment="csi-xxxxxx" err="failed to attach: rpc error: code = NotFound desc = failed to get azure instance id for node \"xxxxxxxx\" (instance not found)"

What you expected to happen:
The disk should be able to attach to the VM in resource group B correctly. I'm not sure how the azuredisk-csi-controller finds the VMs but should it not be getting the resource group of a VM from its providerID?
How to reproduce it:

Anything else we need to know?:

Environment:

• CSI Driver version: 1.31
• Kubernetes version (use kubectl version): 1.30
• OS (e.g. from /etc/os-release): Ubuntu 20.04.6 LTS
• Kernel (e.g. uname -a): 5.4.0-1149-azure-fips
• Install tools:
• Others:

[yangl900]

Some additional notes:

• The target VM is VMSS Flex
• the node name in failed to get azure instance id for node \"xxxxxxxx\" error message is the k8s node name, which does not map to the actual VM name. (not sure if this is expected)

[yangl900]

btw for the node name, for example, the node name looks like c3c394ad3243-dgvdzrsjtjd3a2, but in Azure the VM resource name is c3c394ad3243-dgvdzrsjtjd3a2_958b2726. I don't think we explicitly set the hostname in VMSS flex.

For example, the node name is c3c394ad3243-dgvdzrsjtjd3a2
Provider Id is: azure:///subscriptions/{sub_id}/resourceGroups/{rg_name}/providers/Microsoft.Compute/virtualMachines/c3c394ad3243-dgvdzrsj_958b2726

[sarandia]

@andyzhangx We added --get-nodeid-from-imds but it still returned the node name c3c394ad3243-dgvdzrsjtjd3a2 as the node ID rather than the VM name c3c394ad3243-dgvdzrsj_958b2726. I checked IMDS and it was like:

{"compute":{"azEnvironment":"AzurePublicCloud","customData":"","evictionPolicy":"","isHostCompatibilityLayerVm":"true","licenseType":"","location":"westus","name":"c3c394ad3243-dgvdzrsj_958b2726","offer":"","osProfile":{"adminUsername":"azureuser","computerName":"c3c394ad3243-dgvdzrsjTJD3A2","disablePasswordAuthentication":"true"},"osType":"Linux","placementGroupId":"","plan":{"name":"","product":"","publisher":""},"platformFaultDomain":"0","platformUpdateDomain":"0","priority":"Regular","provider":"Microsoft.Compute","
...
"vmId":"2716d013-40b7-480f-86db-e544ec63f384","vmScaleSetName":"c3c394ad3243-dgvdzrsj","vmSize":"Standard_D4s_v5","zone":""},

And the node csi driver (azuredisk) logs were:

I0528 18:08:14.157608       1 nodeserver.go:408] NodeGetInfo: NodeID(c3c394ad3243-dgvdzrsjtjd3a2), metadata.Compute.Name(c3c394ad3243-dgvdzrsj_958b2726)
E0528 18:08:14.157626       1 nodeserver.go:413] getVMSSInstanceName failed with parsing vmss compute name(c3c394ad3243-dgvdzrsj_958b2726) failed with strconv.Atoi: parsing "958b2726": invalid syntax

We have the following flags to the CSI node driver:

        - --v=5
        - --endpoint=$(CSI_ENDPOINT)
        - --nodeid=$(KUBE_NODE_NAME)
        - --enable-perf-optimization=true
        - --allow-empty-cloud-config=true
        - --get-node-info-from-labels=false
        - --get-nodeid-from-imds=true

Could you help take a look at why it's not getting the correct name from IMDS?
Seems the parsing logic expects the instance ID in a VMSS (after "_") to be an integer but in this case it was 958b2726

[andyzhangx]

this issue was fixed by #3091 in CSI driver v1.31.10, could you try that version? thx

[sarandia]

@andyzhangx I tried 1.30.10 but still got the same error.

I0529 05:02:46.603642       1 azure_zones.go:166] Availability zone is not enabled for the node, falling back to fault domain
I0529 05:02:46.603653       1 nodeserver.go:355] NodeGetInfo, nodeName: c3c394ad3243-dgvdzrsjtjd3a2, failureDomain: 0
I0529 05:02:46.603667       1 nodeserver.go:441] got a matching size in getMaxDataDiskCount, VM Size: STANDARD_D4S_V5, MaxDataDiskCount: 8
I0529 05:02:46.603675       1 nodeserver.go:408] NodeGetInfo: NodeID(c3c394ad3243-dgvdzrsjtjd3a2), metadata.Compute.Name(c3c394ad3243-dgvdzrsj_958b2726)
E0529 05:02:46.603688       1 nodeserver.go:413] getVMSSInstanceName failed with parsing vmss compute name(c3c394ad3243-dgvdzrsj_958b2726) failed with strconv.Atoi: parsing "958b2726": invalid syntax
I0529 05:02:46.603710       1 utils.go:112] GRPC response: {"accessible_topology":{"segments":{"topology.disk.csi.azure.com/zone":""}},"max_volumes_per_node":8,"node_id":"c3c394ad3243-dgvdzrsjtjd3a2"}

looking at your fix, I think in this case if you look at the logs (NodeGetInfo: NodeID(c3c394ad3243-dgvdzrsjtjd3a2), metadata.Compute.Name(c3c394ad3243-dgvdzrsj_958b2726)), the node ID was not empty. I think this is because the node ID is set here: https://github.com/kubernetes-sigs/azuredisk-csi-driver/pull/3091/files#diff-156099288763837ad0c787f45bac8fd4d37c584f17956cbdb0287cadeb845221L419 already so if there's an error parsing the VMSS name for VMSS flex then it won't be nodeID =""

[andyzhangx]

@sarandia you could remove the nodeid config in csi daemonset config, thus it would be empty:

azuredisk-csi-driver/deploy/csi-azuredisk-node.yaml

Line 93 in f69a21c

- "--nodeid=$(KUBE_NODE_NAME)"

[sarandia]

@andyzhangx After removing --node-id, seems it started crashlooping with

I0529 05:15:59.059738       1 azure_vmclient.go:209] Received error in vm.list.request: resourceID: ...providers/Microsoft.Compute/virtualMachines, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../providers/Microsoft.Compute/virtualMachines?api-version=2022-03-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=b7f4740a-8820-48c4-a683-6541d9411e49&resource=https%3A%2F%2Fmanagement.core.windows.net%2F
E0529 05:15:59.059774       1 azure_vmsets_repo.go:62] VirtualMachinesClient.List() failure with err=&{false 400 0001-01-01 00:00:00 +0000 UTC azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../providers/Microsoft.Compute/virtualMachines?api-version=2022-03-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=b7f4740a-8820-48c4-a683-6541d9411e49&resource=https%3A%2F%2Fmanagement.core.windows.net%2F}
E0529 05:15:59.059813       1 azure_vmss.go:534] Failed to check VM management type: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group xxx: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../providers/Microsoft.Compute/virtualMachines?api-version=2022-03-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=b7f4740a-8820-48c4-a683-6541d9411e49&resource=https%3A%2F%2Fmanagement.core.windows.net%2F
W0529 05:15:59.059826       1 nodeserver.go:344] get zone() failed with: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group xxx: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../providers/Microsoft.Compute/virtualMachines?api-version=2022-03-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=b7f4740a-8820-48c4-a683-6541d9411e49&resource=https%3A%2F%2Fmanagement.core.windows.net%2F, fall back to get zone from node labels
I0529 05:15:59.059872       1 request.go:1112] Error in request: resource name may not be empty
E0529 05:15:59.059897       1 utils.go:110] GRPC error: rpc error: code = Internal desc = getNodeInfoFromLabels on node() failed with get node() failed with resource name may not be empty
I0529 05:16:11.816336       1 utils.go:105] GRPC call: /csi.v1.Identity/GetPluginInfo
I0529 05:16:11.816351       1 utils.go:106] GRPC request: {}
I0529 05:16:11.816381       1 utils.go:112] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"v1.31.10"}
I0529 05:16:13.063569       1 utils.go:105] GRPC call: /csi.v1.Node/NodeGetInfo
I0529 05:16:13.063582       1 utils.go:106] GRPC request: {}
I0529 05:16:13.063628       1 azure_vmss_cache.go:335] refresh the cache of NonVmssUniformNodesCache in rg &{map[xxx:{}]}
E0529 05:16:13.069352       1 util.go:139] Send.sendRequest: error in parsing response body string "": unexpected end of JSON input, Skip retrying regional host
I0529 05:16:13.069376       1 azure_vmclient.go:209] Received error in vm.list.request: resourceID: .../providers/Microsoft.Compute/virtualMachines, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../providers/Microsoft.Compute/virtualMachines?api-version=2022-03-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=b7f4740a-8820-48c4-a683-6541d9411e49&resource=https%3A%2F%2Fmanagement.core.windows.net%2F
E0529 05:16:13.069422       1 azure_vmsets_repo.go:62] VirtualMachinesClient.List(xxx) failure with err=&{false 400 0001-01-01 00:00:00 +0000 UTC azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../providers/Microsoft.Compute/virtualMachines?api-version=2022-03-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=b7f4740a-8820-48c4-a683-6541d9411e49&resource=https%3A%2F%2Fmanagement.core.windows.net%2F}
E0529 05:16:13.069460       1 azure_vmss.go:534] Failed to check VM management type: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group xxx: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../providers/Microsoft.Compute/virtualMachines?api-version=2022-03-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=b7f4740a-8820-48c4-a683-6541d9411e49&resource=https%3A%2F%2Fmanagement.core.windows.net%2F
W0529 05:16:13.069469       1 nodeserver.go:344] get zone() failed with: getter function of nonVmssUniformNodesCache: failed to list vms in the resource group xxx: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/.../providers/Microsoft.Compute/virtualMachines?api-version=2022-03-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=b7f4740a-8820-48c4-a683-6541d9411e49&resource=https%3A%2F%2Fmanagement.core.windows.net%2F, fall back to get zone from node labels
I0529 05:16:13.069498       1 request.go:1112] Error in request: resource name may not be empty
E0529 05:16:13.069525       1 utils.go:110] GRPC error: rpc error: code = Internal desc = getNodeInfoFromLabels on node() failed with get node() failed with resource name may not be empty

Note that intentionally we don't give identities to these nodes to call the cloud API

[andyzhangx]

@sarandia I have worked out a PR to fix this issue, are you able to verify with this temp docker image? andyzhangx/azuredisk-csi:v1.34.0 (keep the --nodeid=$(KUBE_NODE_NAME) and only enable --get-nodeid-from-imds=true

[sarandia]

@sarandia I have worked out a PR to fix this issue, are you able to verify with this temp docker image? andyzhangx/azuredisk-csi:v1.34.0 (keep the --nodeid=$(KUBE_NODE_NAME) and only enable --get-nodeid-from-imds=true

Hi @andyzhangx we are not allowed to use an image hosted in an external repository. We'll need the source code branch and build the image ourselves. Can we merge the fix to the 1.31 release branch and we can test it out?

[sarandia]

@sarandia I have worked out a PR to fix this issue, are you able to verify with this temp docker image? andyzhangx/azuredisk-csi:v1.34.0 (keep the --nodeid=$(KUBE_NODE_NAME) and only enable --get-nodeid-from-imds=true

Hi @andyzhangx we are not allowed to use an image hosted in an external repository. We'll need the source code branch and build the image ourselves. Can we merge the fix to the 1.31 release branch and we can test it out?

Actually I was able to use your image but I got the following errors in the CSI disk controller:

I0529 17:50:09.990490       1 csi_handler.go:243] "Error processing" VolumeAttachment="csi-0780facba24782f99d523720650b28a98637c7134f3c114d08ec4d6c9622be5a" err="failed to attach: rpc error: code = NotFound desc = failed to get azure instance id for node \"c3c394ad3243-dgvdzrsj_958b2726\" (instance not found)"

even though the NodeID field in the CSInode object was correct.

I debugged more and found:
We're running into this error which is from this code in the Azure cloud provider lib. This eventually takes us to this call to fetch the VMSS and there's a comment here saying it doesn't support VMSS flex.

[andyzhangx]

@sarandia there is vmType in cloud provider config, what's current value? could you set as vmType: vmssflex?

[sarandia]

@andyzhangx is it the vmType in the cloud provider config of the csi-azuredisk-controller or csi-azuredisk-node?
If it's the controller then the problem is our k8s cluster has both VMSS uniform nodes and VMSS flex nodes.

[andyzhangx]

vmType in the cloud provider config of the csi-azuredisk-controller

it's vmType in the cloud provider config of the csi-azuredisk-controller, could you try vmType: vmssflex first?

[sarandia]

@andyzhangx seems it's still the same error after I set the vmType to vmssflex for the csi-azuredisk-controller