Hello everyone,
I’m facing an issue with a standard service-to-service call between two Cloud Run services in the same project, and I’m hoping for some ideas from the community. I found an old, probably unresolved topic which seems to be related
I’ve made a simple reproducable example for this scenario.
The Goal:
I have a public-facing service (hello-caller) that needs to call a backend service (hello-internal) which is configured with Ingress: internal.
The Problem:
The call consistently fails with a generic 404 Not Found error from the Google Front End. However, if I change the ingress of the internal service to all, the exact same call works perfectly. This proves that the URL and the application code are correct.
The logs of the calling service show that an Authorization header with a bearer token is being successfully attached to the request, so it seems the request is authenticated, but is still being treated as anonymous by the infrastructure.
Minimal Reproducible Example
To rule out any issues in my application code, I’ve created a minimal “Hello World” test case, which fails in the exact same way.
1. The internal service (hello-internal): A simple Express server returning “Hello”.
package.json:
{
"type": "module",
"dependencies": {
"express": "^4.0.0"
}
}
index.js:
import express from "express";
const app = express();
app.get("/", (req, res) => res.send("Hello from internal service!"));
const port = process.env.PORT || 8080;
app.listen(port, () => console.log(`Listening on ${port}`));
Deployed with: gcloud run deploy hello-internal --source . --ingress=internal --allow-unauthenticated --region=europe-west10
2. The public calling service (hello-caller): An Express server that uses google-auth-library to make an authenticated request.
package.json:
{
"type": "module",
"dependencies": {
"express": "^4.0.0",
"google-auth-library": "^9.0.0"
}
}
index.js:
import express from "express";
import { GoogleAuth } from "google-auth-library";
const app = express();
const auth = new GoogleAuth();
const targetUrl = "https://hello-internal-XXXXXXXXXXXX.europe-west10.run.app";
let client;
app.get("/", async (req, res) => {
try {
if (!client) client = await auth.getIdTokenClient(targetUrl);
const response = await client.request({ url: targetUrl });
res.send(`Response from internal service: "${response.data}"`);
} catch (err) {
console.error("ERROR:", err);
res.status(500).send("Failed to call internal service.");
}
});
const port = process.env.PORT || 8080;
app.listen(port, () => console.log(`Listening on ${port}`));
Deployed with: gcloud run deploy hello-caller --source . --ingress=all --allow-unauthenticated --region=europe-west10
Logs
ERROR: GaxiosError:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>404 Page not found</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Page not found</h1>
<h2>The requested URL was not found on this server.</h2>
<h2></h2>
</body></html>
at Gaxios._request (/workspace/node_modules/gaxios/build/src/gaxios.js:142:23)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async IdTokenClient.requestAsync (/workspace/node_modules/google-auth-library/build/src/auth/oauth2client.js:429:18)
at async file:///workspace/index.js:13:26 {
config: {
url: 'https://hello-internal-XXXXXXXXXXXX.europe-west10.run.app/',
headers: {
Authorization: '<<REDACTED> - See `errorRedactor` option in `gaxios` for configuration>.',
'User-Agent': 'google-api-nodejs-client/9.15.1',
'x-goog-api-client': 'gl-node/22.17.1'
},
paramsSerializer: [Function: paramsSerializer],
validateStatus: [Function: validateStatus],
responseType: 'unknown',
method: 'GET',
errorRedactor: [Function: defaultErrorRedactor]
},
response: {
config: {
url: 'https://hello-internal-XXXXXXXXXXXX.europe-west10.run.app/',
headers: [Object],
paramsSerializer: [Function: paramsSerializer],
validateStatus: [Function: validateStatus],
responseType: 'unknown',
method: 'GET',
errorRedactor: [Function: defaultErrorRedactor]
},
data: '\n' +
'<html><head>\n' +
'<meta http-equiv="content-type" content="text/html;charset=utf-8">\n' +
'<title>404 Page not found</title>\n' +
'</head>\n' +
'<body text=#000000 bgcolor=#ffffff>\n' +
'<h1>Error: Page not found</h1>\n' +
'<h2>The requested URL was not found on this server.</h2>\n' +
'<h2></h2>\n' +
'</body></html>\n',
headers: {
'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000',
'content-length': '272',
'content-type': 'text/html; charset=UTF-8',
date: 'Tue, 29 Jul 2025 11:34:52 GMT'
},
status: 404,
statusText: 'Not Found',
request: {
responseURL: 'https://hello-internal-XXXXXXXXXXXX.europe-west10.run.app/'
}
},
error: undefined,
status: 404,
[Symbol(gaxios-gaxios-error)]: '6.7.1'
}
What I’ve Already Ruled Out
I have spent a significant amount of time troubleshooting and can confirm the following:
- IAM Permissions: The
hello-callerservice account has theCloud Run Invokerrole onhello-internalAND theService Account Token Creatorrole on itself. I even temporarily granted theOwnerrole, which did not solve the issue. - Correct Code: The
google-auth-libraryis used as documented. A debug endpoint confirmed it can successfully create anIdTokenClient. - VPC / Networking: There are no VPC Connectors or VPC Service Controls active in this project.
My Question:
Given that even a minimal, correctly configured test case fails, and that others have experienced this exact problem, is there a known platform issue, an organization policy, or a hidden configuration that could be causing this behavior?
Any ideas or suggestions would be greatly appreciated. Thank you!