Cloud DNS geolocation policy with health checking never fails over for internal Application Load Balancer targets - what does "health of the load balancer itself" mean?

[Kacper_Zajkowski]

Hi all,
I’m testing multi-region failover for private endpoints (Apigee X instances behind PSC NEGs) and I can’t get Cloud DNS health checking to fail over when the targets are internal Application Load Balancers. With internal passthrough Network Load Balancers as targets, the same setup works correctly and DNS routes traffic to healthy target.

Setup

• Private zone, A record with GEO routing policy + --enable-health-checking.
• Targets: forwarding rules of internal Application Load Balancers in europe-west1 and us-central1, global access enabled. describe shows healthCheckedTargets with loadBalancerType regionalL7ilb (and globalL7ilb for a cross-region ALB).
• ALB backends: Private Service Connect NEGs pointing to a service attachment in front of an internal passthrough NLB.
• Client VM in europe-west1, querying every 5 s, TTL 30 s.

Diagram 1 - request path with internal ALB + PSC; red boxes = what I broke and what DNS did:

Diagram 2 - control test with internal passthrough NLB ( Internal passthrough Network Load Balancer overview  |  Cloud Load Balancing  |  Google Cloud Documentation ) as DNS target; green = failover worked:

What I tried - in every case DNS kept returning the europe-west1 VIP (10+ min, once 1+ hour):

1. Backend behind PSC stopped (clients get 504).
2. Forwarding rule deleted.
3. Backend removed from the backend service.
4. Whole ALB deleted (FR, target proxy, URL map, backend service) incl. proxy-only subnet.
5. ALB with a VM instance group backend, health check blocked by firewall - get-health shows 100% UNHEALTHY.
6. Cross-region internal ALB as target, forwarding rule in europe-west1 deleted.

Control test: the same record pointing to internal passthrough NLB forwarding rules fails over within seconds once the NLB backends are UNHEALTHY. So the mechanism works! But unfortunately, I cannot use it because of PSC.

The docs only say that for internal ALBs “Cloud DNS considers the health of the load balancer itself” and that backends are handled by the LB.

Questions

1. Which signal does Cloud DNS use for internal ALB targets - only the health of the Google-managed proxy layer in the region?
2. Is it expected that DNS keeps returning the IP of a deleted forwarding rule (no signal = healthy)?
3. Is there any supported way to simulate an unhealthy internal ALB to test DNS failover?
4. What is the expected failover time for internal ALB targets?

Thanks!

[zachs]

Is it expected that Cloud DNS health checking for internal Application Load Balancers (regionalL7ilb / globalL7ilb) never fails over based on backend or PSC NEG problems (5xx, timeout, deleted backend, etc.)?
What does “the health of the load balancer itself, not that of individual backend instances” mean in practice here?

Yes, this is expected behavior. For proxy-based internal load balancers, Cloud DNS monitors the internal health of the Google-managed proxy fleet in your proxy-only subnet - not your backend services, instance groups, or Private Service Connect (PSC) NEGs.

With an internal passthrough Network Load Balancer, there is no proxy layer, so Cloud DNS directly evaluates the health check status of your backend instances or NEGs and fails over when at least 80% of those backends are unhealthy. With an internal Application Load Balancer, the proxies are still up and accepting TCP connections on the forwarding rule IP even when every backend returns 5xx or times out, so Cloud DNS continues to see the load balancer IP as healthy (healthyIps in dns_query logs).

Under what exact conditions can an internal ALB target ever become unhealthy from Cloud DNS’s perspective?

An internal Application Load Balancer target only becomes unhealthy in Cloud DNS if the Google-managed proxy pool in that region loses availability and stops passing internal platform health probes. Customer-side conditions - such as all backends failing health checks, PSC NEGs returning 502/503/504, or firewall rules blocking backend health check probes - won’t mark the forwarding rule unhealthy in Cloud DNS.

Regarding your test where deleting the forwarding rule still returned the old IP, my assumption is that you deleted the forwarding rule while the Cloud DNS record set still referenced it. When a forwarding rule is deleted out from under an existing routing policy record, Cloud DNS stops receiving health signals for that target and fails open (treating no health signal as healthy) rather than blackholing queries. You should remove or replace the target in the Cloud DNS routing policy before deleting the forwarding rule.

If the goal is automatic cross-region failover when one regional private endpoint (PSC NEG / Apigee instance) is down or returning 5xx, what is the Google-recommended pattern for internal DNS + internal LB?

Regional internal Application Load Balancers only support a single PSC NEG per backend service and don’t support cross-region failover. To get automatic cross-region failover across regional Apigee X PSC attachments, use a Cross-Region Internal Application Load Balancer instead of relying on Cloud DNS to switch between isolated regional load balancers:

1. Attach both regional PSC NEGs (europe-west1 and us-central1) to a single global backend service. (If you are testing with your own custom PSC producer service attachment rather than Apigee X, make sure the producer’s internal forwarding rule has --allow-global-access enabled.)
2. Create regional forwarding rules for the Cross-Region Internal Application Load Balancer in each region (europe-west1 and us-central1) and keep your Cloud DNS geolocation policy pointing to those forwarding rules (globalL7ilb) so clients always connect to their closest regional IP.
3. Configure outlier detection on the global backend service so the local proxies automatically eject a PSC NEG that returns consecutive 5xx or gateway errors (502/503/504) and fail traffic over to the healthy region’s PSC NEG. Note that if you customize --outlier-detection-consecutive-errors, you also need to set --outlier-detection-enforcing-consecutive-errors=100:

gcloud compute backend-services update <GLOBAL_BACKEND_SERVICE> \
    --global \
    --outlier-detection-consecutive-errors=3 \
    --outlier-detection-consecutive-gateway-failure=3 \
    --outlier-detection-enforcing-consecutive-errors=100 \
    --outlier-detection-enforcing-consecutive-gateway-failure=100 \
    --outlier-detection-interval=5s \
    --outlier-detection-base-ejection-time=30s \
    --outlier-detection-max-ejection-percent=100

With this setup, Cloud DNS routes the client to the nearest regional proxy IP, and if the europe-west1 PSC NEG starts returning 5xx errors or gateway timeouts, the proxies automatically fail requests over across regions to the us-central1 PSC NEG without waiting on DNS TTLs or DNS health checks.

[Kacper_Zajkowski]

Thanks, that matches my tests. I already tested the cross-region ALB + outlier detection setup:

• backend returning 503 / refusing connections: failover within seconds
• backend hanging (no response): Envoy returns 504 “upstream request timeout”, NEG never ejected

Questions:

1. Should Envoy timeouts count for outlier detection on PSC NEGs? If not, how do you detect a hung backend?
2. How do you recommend failing over the regional entry point (our misconfiguration, deleted FR) if DNS only checks the proxy layer?