Deleted pvc is Terminating forever

Google Cloud Serverless Applications
1

I switched the default storage class from “standard” to “standard-rwo”, and I am facing several issues.

First issue was that helm would not update as the storage class changed. To fix this I manually deleted the pvc.

Now came the second problem. It’s not possible to delete the pvc anymore as it gets stuck in “Terminating” for hours, currently 10 hours or so.

$ kubectl get pv | grep …

pvc-22d389de-7847-4bbe-a7d7-29d68ac55c49 600Gi RWO Delete Terminating some-namespace/staging standard 21d

So how do I delete these pvcs?

$ kubectl describe pv/pvc-22d389de-7847-4bbe-a7d7-29d68ac55c49
Name: pvc-22d389de-7847-4bbe-a7d7-29d68ac55c49
Labels: topology.kubernetes.io/region=europe-west2
topology.kubernetes.io/zone=europe-west2-c
Annotations: kubernetes.io/createdby: gce-pd-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/migrated-to: pd.csi.storage.gke.io
pv.kubernetes.io/provisioned-by: kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pv-protection]
StorageClass: standard
Status: Terminating (lasts 8h)
Claim: some-namespace/staging
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 600Gi
Node Affinity:
Required Terms:
Term 0: topology.kubernetes.io/zone in [europe-west2-c]
topology.kubernetes.io/region in [europe-west2]
Message:
Source:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: gke-dev-m-02-d4e99bca–pvc-22d389de-7847-4bbe-a7d7-29d68ac55c49
FSType: ext4
Partition: 0
ReadOnly: false
Events:

2

It seems like you tried to delete the pv (persistentVolume) without deleting the persistentvolumeclaim(pvc). That’s expected behavior as we have something called volume protection

Can you share the output of the following commands:

kubectl get storageclass

kubectl get pvc

kubectl get pvc