When validating a new region, warming up caches after maintenance, or migrating workloads behind a Global External Application Load Balancer, teams often need a fixed global traffic split - such as 70% in us-south1 and 30% in us-east4 - with automatic failover if either region goes down.
Default load balancing configurations force a trade-off between explicit traffic weights and automatic health-based failover. Configuring URL Map weightedBackendServices across two single-region backend services enforces a strict 70/30 split, but the load balancer selects the target backend service before checking backend health. If the 70% region goes down, 70% of requests fail with HTTP 503. Conversely, putting both regions in a single multi-region backend service provides automatic failover, but routes requests by client proximity instead of a fixed percentage.
This post walks through a deployment pattern that combines URL Map weightedBackendServices with two shared multi-region Backend Services, inverted preferred backends (PREFERRED and DEFAULT), and a ServiceLbPolicy (autoCapacityDrain, failoverHealthThreshold, and SPRAY_TO_REGION) to achieve a deterministic 70/30 split and automatic cross-region failover, along with lab test results and the operational caveats to plan for before deploying.
What it is
A Global External Application Load Balancer evaluates routing in two stages:
- Stage 1 - URL Map route evaluation (
weightedBackendServices): On each request, the load balancer evaluates the URL Map and selects a backend service proportionally based on the configured weights (weight / total_weight). This selection happens before inspecting backend health or capacity. Even if you configure a URL MapretryPolicy, retries are sent to the same backend service rather than selecting a different weighted backend service. - Stage 2 - Backend Service capacity and health routing: Once a backend service is chosen, the load balancer distributes requests across its attached zonal instance groups or Network Endpoint Groups (NEGs) based on health checks, capacity limits, and the attached
ServiceLbPolicy.
This two-stage design creates a dilemma when you need both weighted splitting and automatic failover. If each weighted backend service only contains backends in one region, Stage 2 has no backup capacity when that region fails. If you attach both regions to each backend service with default settings, Stage 2 routes requests to whichever region is closest to the client, overriding your 70/30 split whenever both regions have spare capacity.
The Solution: Attach the same zonal Managed Instance Groups (MIGs), Unmanaged Instance Groups, or zonal NEGs from both regions to two backend services, invert the --preference=PREFERRED and --preference=DEFAULT settings between them, and attach a ServiceLbPolicy with autoCapacityDrain and SPRAY_TO_REGION. The 70%-weighted backend service prefers us-south1 with us-east4 as its fallback, while the 30%-weighted backend service prefers us-east4 with us-south1 as its fallback.
Why it matters
During regional migrations or canary validations, you shouldn’t have to sacrifice regional redundancy to control traffic percentages.
Here is how the inverted preferred-backend pattern compares to standard multi-region options:
| Approach / Method | Enforces 70/30 Split Regardless of Client Location? | Automatic Cross-Region Failover? | Best For |
|---|---|---|---|
URL Map weightedBackendServices (1 region per backend service) |
Yes | No (70% fail with HTTP 503 if the primary region fails) |
Same-region blue/green or canary deployments where regional failover isn’t needed. |
Single multi-region backend service (WATERFALL_BY_REGION + capacityScaler) |
No (Routes to the closest region first until capacity saturates) | Yes | Steady-state active/active deployments optimized for lowest client latency. |
Single multi-region backend service (PREFERRED + DEFAULT) |
100/0 only (All traffic routes to PREFERRED until full or unhealthy) |
Yes | Strict active/passive regional failover or on-premises-to-cloud overflow. |
URL Map weightedBackendServices + 2 shared multi-region backend services with inverted PREFERRED / DEFAULT + ServiceLbPolicy |
Yes | Yes | Regional warm-up, migration validation, or weighted active/active splits requiring automatic failover. |
Key benefits
- Client-location-independent traffic weights: Setting
preference: PREFERREDon a backend overrides proximity routing (WATERFALL_BY_REGION), so the 70% backend service sends 100% of its share tous-south1and the 30% backend service sends 100% of its share tous-east4. - Automatic cross-region failover: If all zones in
us-south1fail health checks, the 70% backend service automatically drainsus-south1and shifts its traffic to itsDEFAULTbackends inus-east4without URL Map or DNS changes. - Balanced intra-region zonal distribution: Combining
autoCapacityDrainwithloadBalancingAlgorithm: SPRAY_TO_REGIONspreads traffic evenly across healthy zones within the preferred region and prevents single-zone failures from leaking traffic cross-region.
How it works
The configuration coordinates URL Map weights, backend preferences, and ServiceLbPolicy capacity draining across steady-state, partial-failure, and full-outage scenarios:
1. Steady-state 70/30 routing and intra-region zone distribution
When all zonal backends in us-south1 (a, b, c) and us-east4 (a, b, c) are healthy, the URL Map sends 70% of requests to bs-south-pref and 30% to bs-east-pref. Inside bs-south-pref, the us-south1 zones are marked PREFERRED, pulling 100% of that 70% share into us-south1 regardless of client geography. Inside bs-east-pref, the us-east4 zones are marked PREFERRED, sending 100% of its 30% share to us-east4.
Although public documentation notes that SPRAY_TO_REGION doesn’t override preferred backend selection across regions, enabling loadBalancingAlgorithm: SPRAY_TO_REGION on the ServiceLbPolicy still applies within the preferred region so traffic is distributed evenly across zones a, b, and c rather than skewing toward one zone at low load.
2. Partial regional failure (1 or 2 zones unhealthy) and auto-capacity draining
For zonal MIGs, Unmanaged Instance Groups, and zonal GKE NEGs, the load balancer calculates a zone’s configured capacity from total configured instances/endpoints rather than healthy ones. Without autoCapacityDrain, losing 1 or 2 zones in a 3-zone primary region leaves the dead zones in the primary schedule and triggers proxy-level failover, leaking traffic to the backup region.
Enabling auto-capacity draining (autoCapacityDrain) sets a zonal backend group’s effective capacity to zero once fewer than 25% of its member instances or endpoints pass load balancer health checks. When 1 or 2 zones in us-south1 fail, autoCapacityDrain removes the unhealthy zones from the load balancing pool and shifts their load to the remaining healthy zone(s) inside us-south1, preserving the global 70/30 split.
3. Complete regional outage and automatic recovery
If all three zones in us-south1 become unhealthy, the URL Map still sends 70% of requests to bs-south-pref. Once health checks mark us-south1 unhealthy and autoCapacityDrain propagates to the load balancer proxies, bs-south-pref routes 100% of its traffic to its healthy DEFAULT backup backends in us-east4. Combined with the 30% on bs-east-pref, us-east4 serves 100% of traffic instead of dropping 70% of requests with HTTP 503.
When us-south1 recovers and 35% or more of its endpoints pass health checks for at least 60 seconds, auto-capacity undraining restores us-south1 capacity and returns traffic to the 70/30 split without dropping active connections.
Test results
To verify how PREFERRED backends, autoCapacityDrain, and SPRAY_TO_REGION behave once health checks and capacity draining converge under failure, we tested a 70/30 URL Map split across six zonal instance groups (three zones a/b/c in the primary region and three zones a/b/c in the secondary region), sending 1,000 HTTPS requests per scenario:
| Failure Scenario (1,000 HTTPS Requests Each, Post-Convergence) | Configuration | Primary Region (a / b / c) |
Secondary Region (a / b / c) |
HTTP 503 Errors |
|---|---|---|---|---|
| 1. Steady State (All 6 zones healthy) | 70/30 + PREFERRED (No ServiceLbPolicy) |
69.5% (11.3% / 31.8% / 26.4%) |
30.5% (3.1% / 4.3% / 23.1%) |
0.0% |
| 1. Steady State (All 6 zones healthy) | 70/30 + PREFERRED + autoCapacityDrain + SPRAY_TO_REGION |
70.3% (23.4% / 23.8% / 23.1%) |
29.7% (10.0% / 10.0% / 9.7%) |
0.0% |
2. Partial Outage: 1 of 3 Primary Zones Down (Zone a unhealthy) |
70/30 + PREFERRED (No ServiceLbPolicy) |
68.1% (0.0% / 49.1% / 19.0%) |
31.9% (2.6% / 10.9% / 18.4%) |
0.0% |
2. Partial Outage: 1 of 3 Primary Zones Down (Zone a unhealthy) |
70/30 + PREFERRED + autoCapacityDrain + SPRAY_TO_REGION |
68.9% (0.0% / 34.0% / 34.9%) |
31.1% (10.4% / 10.4% / 10.3%) |
0.0% |
3. Partial Outage: 2 of 3 Primary Zones Down (Zones a & b unhealthy) |
70/30 + PREFERRED (No ServiceLbPolicy) |
45.3% (0.0% / 0.0% / 45.3%) |
54.7% (14.2% / 9.6% / 30.9%) |
0.0% |
3. Partial Outage: 2 of 3 Primary Zones Down (Zones a & b unhealthy) |
70/30 + PREFERRED + autoCapacityDrain + SPRAY_TO_REGION |
69.8% (0.0% / 0.0% / 69.8%) |
30.1% (9.8% / 9.7% / 10.6%) |
0.1% |
| 4. Full Primary Region Outage (All 3 primary zones unhealthy) | Baseline: 70/30 single-region backend services (with retryPolicy) |
0.0% | 30.0% | 70.0% (503) |
| 4. Full Primary Region Outage (All 3 primary zones unhealthy) | 70/30 + PREFERRED (No ServiceLbPolicy) |
0.0% | 100.0% (31.8% / 18.6% / 49.6%) |
0.0% |
| 4. Full Primary Region Outage (All 3 primary zones unhealthy) | 70/30 + PREFERRED + autoCapacityDrain + SPRAY_TO_REGION |
0.0% | 100.0% (34.5% / 32.6% / 32.9%) |
0.0% |
Three behaviors stand out from the test data:
- Intra-region zonal balance (
SPRAY_TO_REGION): WithoutSPRAY_TO_REGION, defaultWATERFALL_BY_REGIONskewed traffic across zones inside each region (11.3% / 31.8% / 26.4%in the primary region and23.1%of the30.5%secondary share landing in zonecduring steady state;49.1%vs.19.0%with 1 zone down; and49.6%in secondary zonecduring full failover). AddingSPRAY_TO_REGIONdistributed requests evenly across all healthy zones (23.4% / 23.8% / 23.1%in steady state,34.0% / 34.9%with 1 zone down, and34.5% / 32.6% / 32.9%during full failover). - Preventing cross-region leakage during multi-zone partial failures (
autoCapacityDrain): When 2 of 3 primary zones failed withoutautoCapacityDrain, the global split collapsed from 70/30 to45.3% / 54.7%(leaking roughly 25% of global traffic to the secondary region), because the load balancer continued scheduling primary traffic to the unhealthy zones before failing over at the proxy layer. WithautoCapacityDrainenabled, the unhealthy zones drained from the pool and the remaining healthy primary zone absorbed the full 70% share (69.8% / 30.1%). - Eliminating persistent 70%
HTTP 503errors during a regional outage: During a complete primary region outage, single-region weighted backend services continued returningHTTP 503on ~70% of requests for the duration of the outage even with a URL MapretryPolicyconfigured. With shared multi-region backend services, once health checks and auto-draining converged, 100% of traffic shifted to the secondary region with near-zero errors (0.0%post-convergence).
Caveats and operational considerations
While this pattern solves weighted traffic splitting with automatic regional failover, it relies on specific interactions between URL Map routing, backend capacity accounting, and ServiceLbPolicy draining. Make sure your deployment accounts for five real-world caveats before deploying to production:
1. Zone symmetry and the 50% auto-drain safety limit
To prevent a widespread health-check issue from draining every backend at once, auto-capacity draining only drains unhealthy candidate backend groups up to 50% of the total backend instance groups or NEGs attached to a backend service. Because all non-empty backends count toward the denominator of that 50% limit, your backup region must have at least as many zonal backend groups as your primary region.
For example, if us-south1 has 3 zonal MIGs or NEGs (us-south1-a/b/c) and us-east4 has 3 zonal MIGs or NEGs (us-east4-a/b/c), each backend service has 6 total zonal groups and autoCapacityDrain can drain all 3 us-south1 groups (3 of 6 = 50%) during a complete us-south1 outage. If us-south1 had 3 zonal groups and us-east4 only had 2 (5 total groups), draining all 3 us-south1 groups would exceed 50% (3 of 5 = 60%), preventing the third unhealthy us-south1 zone from auto-draining.
2. Independent rate limits (balancingMode: RATE) on shared backend groups
When you attach the same zonal MIG or NEG to two backend services (bs-south-pref and bs-east-pref) that use different preferred-backend settings, the load balancer evaluates maxRatePerInstance or maxRatePerEndpoint per backend service rather than sharing a single combined rate counter across both.
This has two practical consequences when using balancingMode: RATE:
- If you configure
maxRatePerEndpoint = 100 RPSon a shared NEG in both backend services, each backend service can send up to 100 RPS to that NEG on its own (up to 200 RPS combined during failover). - Conversely, if you set
maxRatePerEndpointtoo close to your normal steady-state request rate, routine traffic spikes on the 70% backend service will exceed its rate target and spill over to the 30% region even when all backends are healthy, distorting your 70/30 split.
To avoid unintended spillover, set maxRatePerInstance or maxRatePerEndpoint high enough above normal peak load so that cross-region failover is triggered strictly by health check failures rather than traffic spikes (or use balancingMode: UTILIZATION on VM instance groups, where CPU utilization reflects aggregate load from both backend services).
3. Instantaneous failover jump in the 30% region (3.3x step-up in seconds)
In a 70/30 split, the secondary region (us-east4) normally serves only 30% of global traffic (and only 10% in a 90/10 split). If the primary region (us-south1) suffers a complete outage, the load balancer shifts the remaining 70% of traffic to us-east4 within seconds of health checks failing - a 3.3x instantaneous jump in traffic (or a 10x jump in a 90/10 split).
Health-check failover happens in seconds, whereas GKE Horizontal Pod Autoscaler (HPA), GKE node auto-provisioning, and Compute Engine MIG autoscalers take tens of seconds to several minutes to spin up new pods or VMs. You cannot rely on reactive autoscaling to absorb a sudden regional failover; pre-size and pre-warm your secondary region’s minimum pod/VM counts, GKE node pools, and Cloud NAT IP capacity to handle 100% of peak global load at all times.
4. Two-tier health thresholds and transient failover convergence (failoverHealthThreshold vs. autoCapacityDrain)
When both failover threshold (failoverHealthThreshold, default 70) and autoCapacityDrain are enabled on a ServiceLbPolicy, the load balancer responds to unhealthy endpoints in two distinct stages:
- Failover spillover (
below 70% down to 25% healthy endpoints): When the percentage of healthy endpoints in a zonal group drops belowfailoverHealthThreshold(70%) but remains at or above25%,autoCapacityDraindoes not zero out the zone’s capacity. Instead, the load balancer keeps the zone active and begins shifting traffic to failover backends. - Complete zone drain (
fewer than 25% healthy endpoints):autoCapacityDrainonly sets a zonal group’s effective capacity to zero when fewer than 25% of its member instances or endpoints pass load balancer health checks. For example, in a zone with 4 VMs or pods,1 of 4healthy (25%) remains above the drain threshold, while0 of 4healthy (0%) triggersautoCapacityDrain. When a drained zone recovers, it must maintain35%or more healthy endpoints for at least60 secondsbefore its capacity is restored.
Keep in mind that during the initial seconds of an unexpected outage - while health check probes reach unhealthyThreshold and while autoCapacityDrain capacity updates propagate asynchronously across global load balancer proxies - some in-flight requests can still hit unhealthy endpoints and return HTTP 502/503 before failover converges. Configuring a URL Map retryPolicy (5xx, connect-failure) for idempotent requests helps mask transient errors during that convergence window while autoCapacityDrain removes the unhealthy zones.
5. Compute Engine MIG load-balancing autoscaling restriction
If your backends are Compute Engine Managed Instance Groups (MIGs), Google Cloud’s backend service restrictions specify that you shouldn’t attach an autoscaled MIG to more than one backend service when using the HTTP Load Balancing Utilization autoscaling metric (--scale-based-on-load-balancing). Two or more backend services referencing the same autoscaled MIG can contradict one another unless the autoscaling metric is unrelated to load balancer activity.
This restriction does not affect GKE Horizontal Pod Autoscaler (HPA), Unmanaged Instance Groups, or MIGs configured to autoscale on CPU utilization (--scale-based-on-cpu) or Cloud Monitoring custom metrics. If you use MIGs, autoscale on CPU or custom metrics while running the dual-backend-service split, and consolidate back to a single global backend service once your weighted validation window finishes.
How to implement
The following gcloud commands configure the ServiceLbPolicy, the two shared multi-region backend services across us-south1-a/b/c and us-east4-a/b/c, and the 70/30 URL Map split. The same configuration works with zonal MIGs, Unmanaged Instance Groups, or zonal GKE NEGs (--network-endpoint-group).
Step 1: Create the global ServiceLbPolicy with auto-drain and SPRAY_TO_REGION
Create a global ServiceLbPolicy enabling autoCapacityDrain, failoverHealthThreshold: 70, and SPRAY_TO_REGION:
export PROJECT_ID="my-gcp-project"
gcloud network-services service-lb-policies create slp-auto-drain \
--location=global \
--load-balancing-algorithm=SPRAY_TO_REGION \
--auto-capacity-drain \
--failover-health-threshold=70 \
--project="${PROJECT_ID}"
Step 2: Create the two shared multi-region backend services with inverted preferred backends
Create bs-south-pref (us-south1 PREFERRED, us-east4 DEFAULT) and bs-east-pref (us-east4 PREFERRED, us-south1 DEFAULT), attaching the same six zonal groups and slp-auto-drain to both:
# Primary Backend Service (70% weight): us-south1 PREFERRED, us-east4 DEFAULT
gcloud compute backend-services create bs-south-pref \
--global \
--load-balancing-scheme=EXTERNAL_MANAGED \
--protocol=HTTP \
--port-name=http \
--health-checks=hc-http \
--service-lb-policy=slp-auto-drain \
--project="${PROJECT_ID}"
for z in a b c; do
gcloud compute backend-services add-backend bs-south-pref \
--global \
--instance-group="mig-us-south1-${z}" \
--instance-group-zone="us-south1-${z}" \
--balancing-mode=RATE \
--max-rate-per-instance=100 \
--preference=PREFERRED \
--project="${PROJECT_ID}"
gcloud compute backend-services add-backend bs-south-pref \
--global \
--instance-group="mig-us-east4-${z}" \
--instance-group-zone="us-east4-${z}" \
--balancing-mode=RATE \
--max-rate-per-instance=100 \
--preference=DEFAULT \
--project="${PROJECT_ID}"
done
# Secondary Backend Service (30% weight): us-east4 PREFERRED, us-south1 DEFAULT
gcloud compute backend-services create bs-east-pref \
--global \
--load-balancing-scheme=EXTERNAL_MANAGED \
--protocol=HTTP \
--port-name=http \
--health-checks=hc-http \
--service-lb-policy=slp-auto-drain \
--project="${PROJECT_ID}"
for z in a b c; do
gcloud compute backend-services add-backend bs-east-pref \
--global \
--instance-group="mig-us-east4-${z}" \
--instance-group-zone="us-east4-${z}" \
--balancing-mode=RATE \
--max-rate-per-instance=100 \
--preference=PREFERRED \
--project="${PROJECT_ID}"
gcloud compute backend-services add-backend bs-east-pref \
--global \
--instance-group="mig-us-south1-${z}" \
--instance-group-zone="us-south1-${z}" \
--balancing-mode=RATE \
--max-rate-per-instance=100 \
--preference=DEFAULT \
--project="${PROJECT_ID}"
done
Step 3: Configure the 70/30 URL Map split
Import the URL Map using defaultRouteAction.weightedBackendServices (which replaces the top-level defaultService field):
gcloud compute url-maps import um-global-alb \
--global \
--project="${PROJECT_ID}" \
--quiet <<EOF
name: um-global-alb
defaultRouteAction:
weightedBackendServices:
- backendService: https://www.googleapis.com/compute/v1/projects/${PROJECT_ID}/global/backendServices/bs-south-pref
weight: 70
- backendService: https://www.googleapis.com/compute/v1/projects/${PROJECT_ID}/global/backendServices/bs-east-pref
weight: 30
EOF
Step 4: Verify steady-state distribution and test regional failover
Verify backend health and send a test batch to confirm the 70/30 regional split and even zonal distribution:
gcloud compute backend-services get-health bs-south-pref \
--global \
--project="${PROJECT_ID}" \
--format="table(backend.basename(),status.healthStatus[].instance.basename(),status.healthStatus[].healthState)"
for i in $(seq 1 1000); do
curl -sk https://example.com/ | jq -r '.zone'
done | sort | uniq -c
In steady state, ~70% of responses come from us-south1-a/b/c (~23% per zone) and ~30% come from us-east4-a/b/c (~10% per zone). Once health checks fail and auto-draining converges across us-south1-a/b/c, 100% of responses shift to us-east4-a/b/c (~33% per zone).
Step 5: Adjust weights or transition to a single backend service after validation
You can update weights in place (50/50, 0/100, or 100/0) by re-importing the URL Map without dropping active client TCP/TLS connections. Once your validation window is complete and you’re ready to return to standard proximity-based active/active routing on a single backend service, transition without downtime:
# 1. Reset us-south1 backends on bs-south-pref to DEFAULT preference
for z in a b c; do
gcloud compute backend-services update-backend bs-south-pref \
--global \
--instance-group="mig-us-south1-${z}" \
--instance-group-zone="us-south1-${z}" \
--preference=DEFAULT \
--project="${PROJECT_ID}"
done
# 2. Point the URL Map to bs-south-pref and remove the secondary backend service
gcloud compute url-maps set-default-service um-global-alb \
--default-service=bs-south-pref \
--global \
--project="${PROJECT_ID}"
gcloud compute backend-services delete bs-east-pref \
--global \
--project="${PROJECT_ID}" \
--quiet