Architecture Setup
-
Frontend: Global Application Load Balancer (Custom Domain)
-
Backend: Single Backend Service with Serverless NEGs
-
Compute: Cloud Run deployed in two regions:
asia-southeast1(Singapore) andasia-southeast3(Bangkok).
The Issue I am physically located in Bangkok. When I hit the Global Load Balancer URL, the traffic is consistently being routed to the Singapore (asia-southeast1) Cloud Run service instead of the local Bangkok (asia-southeast3) one.
Troubleshooting Performed so far:
-
Both regions are correctly added as Serverless NEGs to the exact same Backend Service.
-
Tested across 3 different top ISPs in Bangkok. All traffic stubbornly routes to Singapore. I tested out from a VM in Google Cloud Bangkok region and even that is routed to Singapore.
-
Hitting the default
.run.appURLs directly proves the Bangkok endpoint is physically closer and significantly faster. The LB latency matches the Singapore endpoint plus slight proxy overhead.
Latency Test Results (Sample size: 20 requests each)
1. Direct Bangkok Cloud Run URL (asia-southeast3):
Plaintext
PS C:\Users\[Username]\Desktop> .\latency.ps1 -Url "https://[masked-app-hash].asia-southeast3.run.app/openapi.json" -Count 20
Summary
-------
Successful requests: 20/20
Min latency: 89.26 ms
Average latency: 106.11 ms
P50 latency: 103.72 ms
2. Direct Singapore Cloud Run URL (asia-southeast1):
Plaintext
PS C:\Users\[Username]\Desktop> .\latency.ps1 -Url "https://[masked-app-hash].asia-southeast1.run.app/openapi.json" -Count 20
Summary
-------
Successful requests: 20/20
Min latency: 116.62 ms
Average latency: 128.08 ms
P50 latency: 121.8 ms
3. Global Load Balancer URL:
Plaintext
PS C:\Users\[Username]\Desktop> .\latency.ps1 -Url "https://api.[masked-custom-domain].app/openapi.json" -Count 20
Summary
-------
Successful requests: 20/20
Min latency: 127.31 ms
Average latency: 138.2 ms
P50 latency: 133.98 ms
Questions for the Community:
-
Why is the GALB completely ignoring the physically closer, lower-latency NEG in
asia-southeast3? -
Are there any known control-plane limitations with the newer
asia-southeast3region and GALB edge routing? -
Besides IAP or Health Check failures, what GCP-side metrics or configurations could force a load balancer to hard-route 100% of traffic away from a perfectly functioning local Serverless NEG?
Any insights from folks who have debugged multi-region Cloud Run routing would be greatly appreciated!