We used to be on 1.30.5-gke.1443001 and everything was fine.
On 2024-12-11 GCP upgraded us to 1.30.5-gke.1699000, and we saw a massive spike in internal connectivity errors.
On 2024-12-15, GCP further bumped is up to 1.30.6-gke.1125000, after which the problems went away.
However there is no mention of what the regression might’ve been, or any incident reports about it. Can someone from GCP help confirm if that version of GKE had some bug? The linked changelog is just for the Github Kubernetes project, and doesn’t account for the dash versions.
Upon checking the monitoring tool, there’s no major error spike from 1.30.5-gke.1443001 to 1.30.5-gke.1699000, not “massive”. For further investigation, you may reach out to Google Cloud Support.
Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.
Could you share the detailed changelogs of the dash-versions if available? We strongly believe it was related to this k8s upgrade, as after the next one the issue went away automatically.
Specifically, the issue we were facing is a lot of network resets / timeouts when a services within k8s was trying to make an HTTP call (over the “internet”) to a service within the same k8s. If we used a proxy (to force it to exit GCP networking) the problem seemed to go away.