Why are new nodes using an old version.

Google Cloud Serverless Applications
1

I am trying to upgrade my autopilot GKE cluster above 1.29.2-gke.1060000 so that I can use Bursting workloads. I’ve upgraded the cluster itself to 1.29.3-gke.1093000 but the nodes are still at 1.27.8-gke.1067004. I tried cordoning off a node so it would get replaced but the new node was still on the old 1.27.8-gke.1067004 version.

Is there a way to force the nodes to be upgraded past 1.29.2 in autopilot?

2 Likes
2

I’m seeing the same thing :thinking:

Running GKE Autopilot with STABLE release channel.
Control plane is upgraded to v1.30.3-gke.1969001 but the nodes are running v1.29.7-gke.1104000

1 Like
3

I am affected as well, GKE Autopilot:

gcloud container clusters list
NAME                            LOCATION      MASTER_VERSION      MASTER_IP     MACHINE_TYPE  NODE_VERSION        NUM_NODES  STATUS
production-k8s-cluster-ce560b4  europe-west4  1.30.5-gke.1014001  x.x.x.x       e2-medium     1.29.1-gke.1589018  64         RUNNING
1 Like
4

Five months later, and this remains an issue. My control plane auto-updated to v1.30.10-gke.1022000, but pod disruption budgets prevented some nodes from upgrading. However, even after removing the PDBs, the overwhelming majority of my nodes are on v1.30.9-gke.1046000. In fact, while typing this, I have now lost one of the newer nodes!

5

Ignore my previous comment as I misread the valid version number. I opened a support request, and GCP identified an orphaned node pool on an older k8s version. It took a few days, but they eventually removed the orphaned node pool and I was able to enable bursting after re-upgrading my cluster.