Understanding how zonal affinity works along with weighted load balancing for internal Network Load Balancers

Google Cloud Community Articles
, ,
1

Context

The objective of this post is to explain and illustrate how zonal affinity feature works in conjunction with weighted load balancing for Internal Network Load balancers. Together, when configured, then can help achieve limit cross zone traffic to reduce costs, latency and improve performance.

We have a sample application that is spread across two zones. Using node selector and labels, the workload is arbitrarily pinned to specific nodes (node 1, node 2, node 3). I have labeled the nodes using kubectl separately.

image
image1000×181 73 KB

Sample workload

#======= Deployment for Node-1 (deployment-1.yaml)=========

apiVersion: apps/v1
kind: Deployment
metadata:
 name: ilb-deployment-node1
spec:
 replicas: 5
 selector:
   matchLabels:
     app: ilb-deployment
 template:
   metadata:
     labels:
       app: ilb-deployment
   spec:
     containers:
     - name: hello-app
       image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
     nodeSelector:
       node-name: node1


#======= Deployment for Node-2 (deployment-2.yaml)========
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: ilb-deployment-node2
spec:
 replicas: 3
 selector:
   matchLabels:
     app: ilb-deployment
 template:
   metadata:
     labels:
       app: ilb-deployment
   spec:
     containers:
     - name: hello-app
       image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
     nodeSelector:
       node-name: node2

#========= Deployment for Node-3 (deployment-3.yaml) ===========
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: ilb-deployment-node3
spec:
 replicas: 2
 selector:
   matchLabels:
     app: ilb-deployment
 template:
   metadata:
     labels:
       app: ilb-deployment
   spec:
     containers:
     - name: hello-app
       image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
     nodeSelector:
       node-name: node3

---
apiVersion: v1
kind: Service
metadata:
 name: int-lb-svc
 annotations:
   networking.gke.io/load-balancer-type: "Internal"
   networking.gke.io/weighted-load-balancing: "pods-per-node"
spec:
 type: LoadBalancer
 externalTrafficPolicy: Local
# trafficDistribution: PreferClose. #commented to show difference w/o Zonal Affinity intially
 selector:
   app: ilb-deployment
 ports:
 - name: tcp-port
   protocol: TCP
   port: 80
   targetPort: 8080

Note the number of replicas assigned to each deployment. We have 5, 3 and 2 for a total of 10 pods. This is to demonstrate how weighted load balancing works later down in conjunction with Zonal Affinity.

Zonal affinity

I have two client VM’s in us-east1-b and us-east-c

Before enabling zonal affinity (commenting out trafficDistribution) , we can see that requests from a client vm in us-east1b is going to both zones B and C.

1000×305 122 KB

After enabling trafficDistribution, we see that traffic from zone-b goes only to us-east1-b.

1000×234 117 KB

Similarly, from a client in us-east1-c, traffic only goes to us-east1-c.

1000×165 61.4 KB

Failover scenario

So, we have 2 nodes in us-east1-b and 1 in us-east1-c (for demo purposes) that are hosting the workload. We are going to simulate failover by cordoning the node in us-east1-c AND deleting the workloads that are pinned to the problematic node

This failover works because for zonal affinity, the spillover attribute is ZONAL_AFFINITY_SPILL_CROSS_ZONE. Through Kubernetes manifest, when we enable weighted load balancing (to be explained later), the spillover ratio defaults to 0. When this ratio is 0, it allows load balancer to spill over to eligible backends in other zones.

Cordoning will ensure that no future workloads are scheduled in the problematic node.

kubectl cordon gke-compute-cluster-1-com-nodepool-1-b4757c56-o7r9

Once the problematic node goes down , they won’t be in service. So, we will delete the deployment to simulate that.

kubectl delete deploy ilb-deployment-node3

Now, we will initiate the requests from us-east1-c and observe where the traffic goes.

1000×250 127 KB

All traffic is going to us-east1-b which still has healthy deployment and nodes.

Once we uncordon it and run the deployment again, we will see that traffic goes back to us-east1-c.

kubectl uncordon gke-compute-cluster-1-com-nodepool-1-b4757c56-o7r9

1000×156 55.5 KB

Weighted Load Balancing

Weighted Load Balancing is about traffic distribution based on backend capacity (weight) and health. Recall that the workload is distributed across the nodes in 50:30:20 ratio. This is more likely to be conspicuous or visible with larger number of requests.

Before Zonal Affinity was enabled, Weighted load balancing was still functional. You will note that relatively speaking, node 1 (50%) received more requests than other two nodes. The one with fewest weight (node 3) which had only 2 replicas received fewest requests.

After zonal affinity was enabled, you will note that zonal affinity is evaluated first and the traffic is distributed according to the weights of the eligible pods/ nodes in that zone. As you can see, this can result in uneven distribution , but that is in line with how zonal affinity + spillover ratio was designed to be. If there are no healthy Pods in the zone, GKE routes traffic to another zone. This implementation optimizes for latency and cost.

3 Likes