GKE standard cluster - pods creation failure - more than one default storage class was found

Google Cloud Serverless Applications
1

Hi,
I had a GKE cluster with a stateful set containing 3 pods with a persistent volume claim each. The pods were torn down by GKE on Jul 27th and they were not created successfully. I was able to notice the following error in the cluster audit log -

"persistentvolumeclaims ""data-<pod name>"" is forbidden: Internal error occurred: 2 default StorageClasses were found" 

I did not change the storage class annotation, storageclass.kubernetes.io/is-default-class for any of the existing storage classes in the cluster. I don’t understand how there were 2 default storage classes suddenly. It would be very helpful to understand how it happened.

To facilitate the pod creation, I changed the storageclass.kubernetes.io/is-default-class annotation from true to false for one of the storage classes. I picked the one that wasn’t supposed to be marked true by comparing it with another healthy cluster of mine. Although, the PVCs and the pods were successfully created, the data from my old persistent volumes were lost. I had to redeploy my stateful set and reconfigure.

2

Hi @Kanipalanisamy

Per my investigation, it seems that this is a known error with 2 default Storage classes found. In this link, there was concern raised way back in 2017. And in this link, there was a discussion on how to fix it, as per thockin choosing either one of the 2 default storage class is fine to resolve the concern.

1 Like
3

Thanks @dionv . It is good to know about the resolution.
May I ask if there was any rollout on the GKE side that could have ended up causing 2 default storage classes in my cluster? My cluster is on stable release channel running 1.25.10-gke.1200 right now but I don’t know the version it was on before the GKE maintenance on Jul 27th.