GKE AutoPilot cluster exceeding Persistent Disk quota

Google Cloud Serverless Applications
1

Hello community,

I’m using a simple GKE Cluster in AutoPilot mode. Autopilot provisions 4 nodes in my case.

Recently I started encountering

Can’t scale up due to exceeded quota

errors.

Using

gcloud compute regions describe europe-west10

I found that the exceeding quota is

- limit: 500.0
  metric: SSD_TOTAL_GB
  usage: 410.0

This is surprising because I don’t use any pd-ssd in my project.

However, after further investigation

kubectl get nodes -o custom-columns=NAME:.metadata.name,STORAGE:.status.capacity.ephemeral-storage

NAME                                               STORAGE
gk3-cluster-name-pool-2-0cdf9a62-cvqf   98831908Ki
gk3-cluster-name-pool-2-0cdf9a62-jm47   98831908Ki
gk3-cluster-name-pool-2-1cf90ee9-4rh5   98831908Ki
gk3-cluster-name-pool-2-264f79fa-pd9b   98831908Ki

It looks like each of the 4 nodes comes with a 100GB ephemeral boot disk which counts towards the quota.

I’m not really sure how to continue from here. I don’t think there is a way to configure the Autopilot nodes to come with a smaller boot disk?

Any ideas?

2

Autopilot uses PD-SSD for the boot disk.
Have you requested a quota increase?

3

Hey @garisingh

No, I haven’t requested a quota increase yet.

My setup is quite simply, so I was surprised to learn that running this project already requires a quota increase. I was wondering if this might related to some misconfiguration on my side that lead the Autopilot boot disk to be provisioned with larger size than required. But from what I understand boot size disk cannot be customised.

So it look to me like increasing the quota is the correct way forward, thanks for the hint!

4

No problem. For new accounts, the base quotas are pretty low, but once you make your quota in crease request, it should be a pretty quick/automatic approval.