Cloud Run service-to-service call fails with 404 for internal service, despite auth token

Hello everyone,

I’m facing an issue with a standard service-to-service call between two Cloud Run services in the same project, and I’m hoping for some ideas from the community. I found an old, probably unresolved topic which seems to be related

I’ve made a simple reproducable example for this scenario.

The Goal:

I have a public-facing service (hello-caller) that needs to call a backend service (hello-internal) which is configured with Ingress: internal.

The Problem:

The call consistently fails with a generic 404 Not Found error from the Google Front End. However, if I change the ingress of the internal service to all, the exact same call works perfectly. This proves that the URL and the application code are correct.

The logs of the calling service show that an Authorization header with a bearer token is being successfully attached to the request, so it seems the request is authenticated, but is still being treated as anonymous by the infrastructure.

Minimal Reproducible Example

To rule out any issues in my application code, I’ve created a minimal “Hello World” test case, which fails in the exact same way.

1. The internal service (hello-internal): A simple Express server returning “Hello”.

package.json:

{
    "type": "module",
    "dependencies": {
        "express": "^4.0.0"
    }
}


index.js:

import express from "express";
const app = express();
app.get("/", (req, res) => res.send("Hello from internal service!"));
const port = process.env.PORT || 8080;
app.listen(port, () => console.log(`Listening on ${port}`));

Deployed with: gcloud run deploy hello-internal --source . --ingress=internal --allow-unauthenticated --region=europe-west10

2. The public calling service (hello-caller): An Express server that uses google-auth-library to make an authenticated request.

package.json:

{
    "type": "module",
    "dependencies": {
        "express": "^4.0.0",
        "google-auth-library": "^9.0.0"
    }
}

index.js:

import express from "express";
import { GoogleAuth } from "google-auth-library";
const app = express();
const auth = new GoogleAuth();

const targetUrl = "https://hello-internal-XXXXXXXXXXXX.europe-west10.run.app";
let client;

app.get("/", async (req, res) => {
    try {
        if (!client) client = await auth.getIdTokenClient(targetUrl);
        const response = await client.request({ url: targetUrl });
        res.send(`Response from internal service: "${response.data}"`);
    } catch (err) {
        console.error("ERROR:", err);
        res.status(500).send("Failed to call internal service.");
    }
});

const port = process.env.PORT || 8080;
app.listen(port, () => console.log(`Listening on ${port}`));


Deployed with: gcloud run deploy hello-caller --source . --ingress=all --allow-unauthenticated --region=europe-west10

Logs
ERROR: GaxiosError: 
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>404 Page not found</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Page not found</h1>
<h2>The requested URL was not found on this server.</h2>
<h2></h2>
</body></html>
 at Gaxios._request (/workspace/node_modules/gaxios/build/src/gaxios.js:142:23)
 at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
 at async IdTokenClient.requestAsync (/workspace/node_modules/google-auth-library/build/src/auth/oauth2client.js:429:18)
 at async file:///workspace/index.js:13:26 {
 config: {
 url: 'https://hello-internal-XXXXXXXXXXXX.europe-west10.run.app/',
 headers: {
 Authorization: '<<REDACTED> - See `errorRedactor` option in `gaxios` for configuration>.',
 'User-Agent': 'google-api-nodejs-client/9.15.1',
 'x-goog-api-client': 'gl-node/22.17.1'
 },
 paramsSerializer: [Function: paramsSerializer],
 validateStatus: [Function: validateStatus],
 responseType: 'unknown',
 method: 'GET',
 errorRedactor: [Function: defaultErrorRedactor]
 },
 response: {
 config: {
 url: 'https://hello-internal-XXXXXXXXXXXX.europe-west10.run.app/',
 headers: [Object],
 paramsSerializer: [Function: paramsSerializer],
 validateStatus: [Function: validateStatus],
 responseType: 'unknown',
 method: 'GET',
 errorRedactor: [Function: defaultErrorRedactor]
 },
 data: '\n' +
 '<html><head>\n' +
 '<meta http-equiv="content-type" content="text/html;charset=utf-8">\n' +
 '<title>404 Page not found</title>\n' +
 '</head>\n' +
 '<body text=#000000 bgcolor=#ffffff>\n' +
 '<h1>Error: Page not found</h1>\n' +
 '<h2>The requested URL was not found on this server.</h2>\n' +
 '<h2></h2>\n' +
 '</body></html>\n',
 headers: {
 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000',
 'content-length': '272',
 'content-type': 'text/html; charset=UTF-8',
 date: 'Tue, 29 Jul 2025 11:34:52 GMT'
 },
 status: 404,
 statusText: 'Not Found',
 request: {
 responseURL: 'https://hello-internal-XXXXXXXXXXXX.europe-west10.run.app/'
 }
 },
 error: undefined,
 status: 404,
 [Symbol(gaxios-gaxios-error)]: '6.7.1'
 }

What I’ve Already Ruled Out

I have spent a significant amount of time troubleshooting and can confirm the following:

  • IAM Permissions: The hello-caller service account has the Cloud Run Invoker role on hello-internal AND the Service Account Token Creator role on itself. I even temporarily granted the Owner role, which did not solve the issue.
  • Correct Code: The google-auth-library is used as documented. A debug endpoint confirmed it can successfully create an IdTokenClient.
  • VPC / Networking: There are no VPC Connectors or VPC Service Controls active in this project.

My Question:

Given that even a minimal, correctly configured test case fails, and that others have experienced this exact problem, is there a known platform issue, an organization policy, or a hidden configuration that could be causing this behavior?

Any ideas or suggestions would be greatly appreciated. Thank you!

As you’ve observed, removing --ingress=internal makes everything work which is expected, since that setting is highly restrictive, as explained in the documentation:

Shared VPC ingress: the Shared VPC network that your revision is configured to send traffic to. For information about when Shared VPC traffic is recognized as “internal”, see Special considerations for Shared VPC.

The following Google Cloud products, if they’re in the same project or VPC Service Controls perimeter as your Cloud Run service and if they’re using the default run.app URL and not a custom domain: [services list not including Cloud Run]

Requests from these sources stay within the Google network, even if they access your service at the run.app URL. Requests from other sources, including the internet, cannot reach your service at the run.app URL or custom domains.

hello-internal can’t be reached via its run.app URL because it’s not exposed externally. To allow communication between services, you need to set up a private connection. See Receive requests from other Cloud Run resources or App Engine.

Thank you so much. That helped me a lot.

I didn’t realised before, that Cloud Run services are not part of that internal restriction in contrary to like Cloud Tasks, Pub/Sub and more.

So I either have to create a VPC connector or set it to --ingress=all.