Failed to update node pool

Google Cloud Serverless Applications
1 
gcloud container clusters upgrade '<my-cluster>' --zone '<zone>' --image-type 'COS_CONTAINERD' --node-pool 'default-pool' --cluster-version '1.24.14-gke.2700'

The output for the operation

All nodes in node pool [default-pool] of cluster [] image will change from COS to COS_CONTAINERD. This operation is long-running and will block
other operations on the cluster (including delete) until it has run to completion.

Do you want to continue (Y/n)? Y

Upgrading my-cluster…done.
ERROR: (gcloud.container.clusters.upgrade) Operation [<Operation
clusterConditions: [<StatusCondition
canonicalCode: CanonicalCodeValueValuesEnum(INTERNAL, 14)
message: ‘Failed to update node pool’>]
detail: ‘Failed to update node pool’
endTime: ‘2023-09-30T07:41:25.517445713Z’
error: <Status
code: 13
details:
message: ‘Failed to update node pool’>
name: ‘operation-’
nodepoolConditions:
operationType: OperationTypeValueValuesEnum(UPGRADE_NODES, 4)
selfLink: ‘https://container.googleapis.com/v1/projects//locations//operations/operation-’
startTime: ‘2023-09-30T07:40:12.108792611Z’
status: StatusValueValuesEnum(DONE, 3)
statusMessage: ‘Failed to update node pool’
targetLink: ‘https://container.googleapis.com/v1/projects//locations//clusters/my-cluster/nodePools/default-pool’
zone: ‘’>] finished with error: Failed to update node pool

]

cluster version: 1.24.14-gke.2700
 node version: v1.23.17-gke.6800
also tried following methods
operation via GUI- same error
 disabled cluster autoscaler and tried upgrade - same error
tried to update the Image type and node version individually - same error
2 Likes
2

Don’t mean to dogpile, but I also am having the same(?) error. I managed to successfully upgrade 1 node pool and the cluster. But any attempt on the remaining node pools fails with this vague error.

I have the same exact cluster and node pool versions. I also tried all the combinations above, all failed.

1 Like
3

Okay, I managed to solve this by creating a brand new node pool using the containerd image and then migrating the workloads to the new node pool. (See https://cloud.google.com/kubernetes-engine/docs/tutorials/migrating-node-pool#step_4_migrate_the_workloads)

1 Like