Private Cloud Run services report Ready but return Google-branded 404s on assigned URLs

I’m troubleshooting two newly created private Cloud Run services. Both report healthy deployment status, but requests to their assigned URLs return the same Google-branded HTML 404 page.

Verified configuration

  • The Cloud Run v2 API reports Ready, RoutesReady, and ConfigurationsReady succeeded.
  • Traffic is assigned 100% to the ready revision.
  • Default URLs are enabled, and ingress is set to ALL.
  • IAM invoker enforcement remains enabled. There are no public invoker grants.
  • The testing user has run.routes.invoke, confirmed using testIamPermissions.
  • Both deployed images are pinned by immutable digest, and both application handlers define GET /healthz.
  • Application data access and operational features remain disabled.

Observed behavior

A single bounded diagnostic made six requests across the two services:

  • Anonymous GET /healthz requests to both Google-assigned URL formats.
  • Authenticated GET /healthz requests to each service’s hashed default URL, using a developer identity token obtained through gcloud.

Every request returned the same Google-branded HTML 404 response, including the authenticated requests. We have not treated this as proof that authentication is working correctly or that the requests reached the containers.

Logging limitation

An initial request-log query returned no entries. Later searches were incomplete: one returned a continuation token, and a targeted search encountered HTTP 500 INTERNAL while following pagination. Consequently, we have not ruled out effective network, ingress, organization-policy, or VPC Service Controls restrictions.

Questions

  1. Which read-only checks would distinguish an effective policy restriction or identity-token issue from a problem with the assigned routes?
  2. What evidence would confirm whether a request reached the Cloud Run frontend or the container?
  3. Are there specific service fields or log filters worth inspecting next?

We want to preserve private invocation and avoid changing permissions or redeploying before understanding the cause.

Project identifiers, service names, actual URLs, account details, and credentials have been omitted.

Additional read-only checks completed:

  • Resource Manager reports no parent organization or folder for this standalone project, so there is no inherited organization/folder policy.
  • The HttpIngress policy-log query completed for the original diagnostic window with no matching entries. The earlier request-log search remains incomplete after HTTP 500 and was not retried.
  • The built-in Cloud Run request_count query completed for both services over the original diagnostic window plus two minutes on each side, with no matching time series. Absence alone is not proof that no frontend handled a request.
  • Fresh service metadata still shows both services Ready, the original immutable images and assigned URLs, ingress ALL, default URLs enabled, IAM invocation enforcement enabled, and application capabilities disabled.

With the owner’s explicit approval, we then made one fresh authenticated GET /healthz to the console service on October 16 at 19:19 UTC. It again returned Google’s HTML 404 page (1,568 bytes, byte-identical to the original response), rather than the application’s JSON health response. We stopped at that first failure; the planned observer request was not made. There were no automatic retries.

We have not changed permissions, disabled authentication, redeployed, or enabled data access. The root cause remains unconfirmed.

Is there another read-only way to distinguish an assigned-route registration problem from a frontend rejection? If a Google engineer needs project-specific routing evidence, what private handoff path is available to an account without a technical support plan?

Second-origin diagnostic update — October 16

We have now made one authenticated GET /healthz from Google Cloud Shell to the same console service’s hashed Google-assigned default URL, using the existing developer identity and an identity token obtained with gcloud auth print-identity-token.

  • HTTP status: 404
  • Content type: text/html
  • Body size: 1,568 bytes
  • Response SHA-256: c2076d92bf1fa30a8ac80cd72fea16056c06f545fd856660c8c34556d10478b0

The response was byte-identical to the earlier Google-branded HTML 404 returned from Windows. The expected application JSON health response did not appear.

This was one request, with redirects and retries disabled, a 20-second HTTP deadline and a 16-KiB response limit. The identity token stayed in process memory. No credentials or request headers are included here.

The preceding configuration checks passed for the assigned URL, expected immutable image, readiness, IAM invoker enforcement and disabled application capabilities. A service IAM policy read also confirmed no public principal bindings. We made no permission, service configuration, image or deployment changes.

Reproducing the same response from a Google-hosted client makes a Windows-only caller issue less likely. It does not prove correct effective authentication, request arrival at the container, or a Google routing incident. The earlier incomplete request-log searches remain a limitation; effective policy and VPC Service Controls restrictions have not been ruled out.

Which read-only frontend, routing or policy evidence would be most useful next? Is there a private handoff path for project-specific evidence when the account has no technical support plan?

Regional service/route metadata update — October 16 (Eastern)

We completed one bounded, read-only check through the regional Cloud Run v1 Admin API for the two affected private services. All four GET requests — the Service and Route resource for each service — returned HTTP 200 with JSON.

  • All four resources report Ready=True, with observed generation matching desired generation (1).
  • Each Route advertises its expected Google-assigned default URL.
  • Both Routes assign 100% of traffic to the expected ready revision. Short and fully qualified revision names were normalized before comparison.
  • Both Service resources show the expected immutable image, and their ingress and ingress-status annotations are both all.
  • Application operational capabilities remain disabled.

Separately, one new request/policy-log query for the earlier Cloud Shell test window (October 16, 20:31–20:34 UTC) completed with zero matching Cloud Run console request-log or HttpIngress policy-log entries, without a continuation token. The earlier incomplete searches were not repeated and remain incomplete. Zero matches in this bounded query do not prove that no frontend or container handled the request.

No new health request, permission change, service update, deployment or key creation was performed. The last health response remains the previously reported Google-branded HTML 404.

The regional Service and Route metadata are consistent with the v2 configuration. These control-plane observations do not establish successful request authentication or endpoint access, and the cause remains unconfirmed.

With the actual regional Route resources also reporting Ready and the expected URL/traffic assignment, which read-only evidence would help distinguish a serving-side routing issue from an effective authentication or policy restriction? Is there a private trace/escalation path for project-specific evidence when the account has no technical support plan?

Project identifiers, service names, actual URLs, account details and credentials remain omitted.

I think it’s the path, not your config. Cloud Run reserves some URL paths that end in “z”, and /healthz is one of them. Requests to those paths get answered by Google’s frontend with that generic HTML 404 and never reach your container. That also fits with you finding nothing in the request logs.

It’s on the Cloud Run known issues page under “Reserved URL paths”. Google recommends avoiding any path that ends in z.

Quick read-only check: send the same authenticated request to another route on the service, like / or a path your app doesn’t have. If you get your app’s own response or its own 404 back, routing and auth are fine. Then rename the health endpoint to /health or /status and point your probes there. No IAM changes needed.

Thank you! The authenticated request to / returned our app successfully. Really appreciate the help!

Glad that was it. If you keep a health check, renaming it to something like /health or /ready gets you out of the reserved paths. And if you can mark the answer as the solution, the next person who runs into the same 404 will find it faster.

Thank you. And solution has been marked. I appreciate your help with this.