I made a new GKE cluster with CSI driver on 1.30.5-gke.1014000. When I created a PVC it told me that the entire europe-west4-c zone was exhausted for SSD PVCs which is not true because I can provision them elsewhere in other projects. I then noticed that my SSD Quota keeps increasing despite there being no new disks. I increated it to 2TB then it filled up. I then increased it to 3 TB, and it again filled up. I have no idea how this is happening as there are 0 disks provisioned except for the node disks themselves that are 128GB each x 6 for 768GB.
I deleted ALL PVCs now in the cluster and all PVs even if they were in Pending status and never provisioned. Does anyone have any ideas? I am wondering if disabling metrics for the cluster could cause this? That’s the only thing I can think of. I am at a loss.
Hi @gcpfan420 ,
Welcome to Google Cloud Community!
The error message indicates that the exhaustion of SSD PVCs occurs when you try to request resources in a zone that cannot accommodate your request due to the current unavailability in the zone.
Try one of the following:
- If you’re requesting a single VM, request the VM in another zone.
- If you’re requesting multiple VMs, try one or more of the following:
- Request the VMs in another zone.
- If your workload permits, request a zonal or regional SSD persistent disk instead of local SSD.
- Split your local SSD request into multiple VMs. For example, if you requested one VM with 600GB of local SSD, divide the request into several VMs that have a total combined local SSD of 600GB.
- You can also check the current status of PVC and PV if it is fully deleted by running commands
kubectl get pvc and kubectl get pv.
For the increasing SSD Quota you may check this some sort of caching related delay on Stack Overflow. As part of their resolution, there was some sort of caching related delay or highlighting the responsible deallocating clusters and disk space that take hours to complete and update the quota usages.
I hope the above information is helpful.