Cannot attach PD to multiple pods in GKE 1.22+

Google Cloud Serverless Applications
1

Hey, I have the following problem: using PVs in ReadOnlyMany access mode does not work for me with newer than 1.21 GKE versions.

I have the following setup: a single PD is used with multiple PVs/PVCs (in various namespaces), and multiple pods in each namespace are attaching to them.
The setup seems to work just fine if my cluster resources (node, control plane etc) use 1.21, but with newer versions (tested multiple) I get the following error from the 2nd pod that is trying to attach the volume:

AttachVolume.Attach failed for volume “my-pv-rox” : rpc error: code = Internal desc = unknown Attach error: failed when waiting for zonal op: operation operation-1656676871696-5e2bd2471f0aa-732b3b98-eb66822e failed (RESOURCE_IN_USE_BY_ANOTHER_RESOURCE): The disk resource ‘projects/dev/zones/europe-west4-a/disks/pd-datasets-v2’ is already being used by 'projects/dev/zones/europe-west4-a/instances/gke-prefect-poc-west-n1-standard-2-co-50ba3e0b-2o7q’I could not find anything related in the release notes. Happy to share my PV/PVC manifests if that helps. (edited)

2

Could you confirm that these requirements are configured in your cluster in order to use persistent disk with multiple readers and pods:

  • Your cluster must run GKE version 1.22 or later.
  • Your cluster must use the Compute Engine persistent disk CSI driver.

Also, please check after the gke upgrade your disk CSI driver version, you can do it this way:

Go to one of the pods (not the daemonset, scroll down and click on of the pdcsi-node-XXXX), you will find it has an annotation that says the component version like this:

components.kge.io/component-version:0.11.6

If after the cluster upgrade, you are still facing the same issue, please manually upgrade the cluster control panel with the following command:

gcloud container clusters upgrade <cluster-name> --master

This will upgrade the pdcsi component version : 0.11.6 and the gcp-compute-persistent-disk-csi-driver:v1.5.1