Tried Sidecar container feature on 2/12 on a RAPID release channel cluster. Was working as expected but tried reproducing it today on all 3 k8s versions in RAPID channel: 1.29.1-gke.1016000 , 1.29.1-gke.1425000 & 1.29.0-gke.1381000 but getting error:
18 Pod has a restartable init container and the SidecarContainers feature is disabled, 1 node(s) had untolerated taint {cloud.google.com/gke-quick-remove: true}
Has this been disabled starting 2/13?
I want to use the SidecarContainers feature in REGULAR release channel for our prod cluster since 1.29.0-gke.1381000 is now available in this channel .
Cmd: kubectl get --raw /metrics | grep kubernetes_feature_enabled
does show:
kubernetes_feature_enabled{name=“SidecarContainers”,stage=“BETA”} 1
Very strange behavior since it was working yesterday. Please advise.
Thanks
Please note that Rapid Channel can give you latest features available and capabilities but, subject for known issues. Try considering to re-create the cluster with newer version or contacting support team for futher investigation.
Thanks for the detailed response @SergeyKanzhelev . We tried the feature in the REGULAR channel this week and it works as expected. Maybe this was a transient issue..
@SergeyKanzhelev All nodes were on the same 1.29.4-gke.1043002 version. Correct, the pod is stuck in the initialization stage forever. The sidecar container had status running and ready false forever. The main container was waiting for the sidecar container to become ready. Initially, I had a readiness probe defined on the sidecar container together with restartPolicy Always. GKE UI showed an error saying the readiness probe could only be defined on init containers with restartPolicy Always. Hence, I mentioned that the restartPolicy setting was ignored. One of the clusters had a Berglas mutation webhook, but the other one did not; that made no difference, it seems.
When you look in cloud console and check for Pod’s yaml, does it show restartPolicy:Always on init container there? If you do kubectl describe pod (use a fresh version of kubectl) - does it show this field?
Do you have any other webhooks installed in your cluster that might have updated the Pod Spec by stripping this field? Maybe istio or something like this? It is not uncommon if the webhook is compiled against the old k8s API.
Just confirmed I was able to get sidecar working on that exact version of GKE cluster. I cannot discount that there may be some GKE settings that can make sidecar broken, but based on all the work and usage we have, I wouldn’t think of this as a likely issue. There might be some third party tooling you use incompatible with the sidecars. I would be interested to learn what it is so we can think of ways to improve experience.
@SergeyKanzhelev Thanks for providing the steps to debug further and confirming that it worked for you. You were right about a webhook stripping the field. I discovered that both clusters had Berglas webhook installed. After disabling it, the example you provided worked fine. Berglas logs also confirmed that. Further looking into Berglas source code, I saw it was compiled against 0.27.2 API [ref] , which, as you mentioned, is where the problem could be. I will look into rebuilding it with a more recent version of K8s API.