GKE Autopilot default-backend-service UNHEALTH

Google Cloud Serverless Applications
1

In my GKE Autopilot environment I am getting a UNHEALTHY status for the backend service (default one) when using GCE ingress (did not defined any defaultBackend in the spec, so it’s suppose to use the GKE default).

Ingress status

annotations:
  ingress.kubernetes.io/backends: {"k8s-be-32569--9510cf09e67605b4":"UNHEALTHY","k8s1-9510cf09-supply-api-gateway-8080-3b6ad708":"HEALTHY"}

If I check in Google console about the details in the k8s-be-32569–9510cf09e67605b4 I get:

{
  "kubernetes.io/service-name":"kube-system/default-http-backend",
  "kubernetes.io/service-port":"\u0026ServiceBackendPort{Name:http,Number:0,}"
}

The kube-system service seems to be correctly working and pointing to l7-default-backend-676d84669b-pngkf

Name:                     default-http-backend
Namespace:                kube-system
Labels:                   addonmanager.kubernetes.io/mode=Reconcile
                          k8s-app=glbc
                          kubernetes.io/cluster-service=true
                          kubernetes.io/name=GLBCDefaultBackend
Annotations:              components.gke.io/layer: addon
Selector:                 k8s-app=glbc
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.24.19.166
IPs:                      10.24.19.166
Port:                     http  80/TCP
TargetPort:               8080/TCP
NodePort:                 http  32569/TCP
Endpoints:                10.24.128.122:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

l7-default-backend-676d84669b-pngkf 1/1 Running 0 25h 10.24.128.122   gk3-autopilot-cluster-nap-1wxbrz3q-c9955dc7-qik2

Looks like the BackendService was wrongly configured for the kubernetes.io/service-port.

Any clue of what is going on here? Did you suffered from a similar issue?

Thanks in advance :slightly_smiling_face:

2

Hi,

With concern that is having an unhealthy backend service from ingress I believe what we need to consider is the liveness and readiness configuration from setting up health check. Let me share this link[1] that can give you more information and gives you the best practice when setting up a health check.

[1]https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes

3

Hi!

Thanks for the quick response. But I am affraid that I followed the best practice recomendation you advise me.

If you check the ingress status I shared

annotations:
ingress.kubernetes.io/backends: {"k8s-be-32569--9510cf09e67605b4":"UNHEALTHY","k8s1-9510cf09-supply-api-gateway-8080-3b6ad708":"HEALTHY"}

k8s1-9510cf09-supply-api-gateway-8080-3b6ad708 : Refers to my backend services and it’s HEALTHY

k8s-be-32569–9510cf09e67605b4 : Refers to the default backend service (that should be provided by the GKE Autopilot env), the one for the 404 responses. And that is the one which is UNHEALTHY

Indeed if I deploy my own l7-default-backend and point to it in the ingress definition, like:

spec:
  defaultBackend:
    service:
      name: default-http-backend
      port:
        number: 80

Then I have everything HEALTHY and working as expected.

Maybe I am missing other things. But not the correct readiness and liveness probes definition.

Thanks for the help.
Berto