Persistent Cloud Run 401 Unauthorized Despite run.invoker Role and Extensive Troubleshooting

Google Cloud Serverless Applications
1

Hi Community,

I’m facing a persistent and baffling issue accessing a private Cloud Run service and would appreciate any insights, as I seem to have exhausted the standard troubleshooting paths.

Goal: Successfully invoke a deployed private Cloud Run service (requires authentication) using my authenticated Google Cloud user account via standard methods like curl with a gcloud access token.

Problem: After deploying the Cloud Run service configured to require authentication, attempts to access the service URL fail:

  1. Browser Access: When logged into Google (tested in Incognito window), accessing the service URL results in 403 Forbidden.

  2. curl with Access Token: Running the following commands in gcloud (authenticated):

    Consistently results in a 401 Unauthorized response page with the text “Error: Forbidden Your client does not have permission to the requested URL / from this server.”

Troubleshooting Steps Performed:

  1. Deployment: Confirmed services deploy successfully via gcloud run deploy … without the --allow-unauthenticated flag (answered ‘N’ to the prompt). Container logs show successful startup.
  2. IAM run.invoker Role: Confirmed via gcloud run services get-iam-policy my-cloud-run-service --region=us-east1 that user definitively has the roles/run.invoker role granted on the service.
  3. Unauthenticated Access: Confirmed that attempting --allow-unauthenticated fails with a FAILED_PRECONDITION error, indicating an Organization Policy correctly prevents public access via allUsers. This is expected.
  4. IAP: Confirmed via Cloud Console (Service → Security tab) that Identity-Aware Proxy (IAP) is Disabled for the service.
  5. IAM Deny Policies: Granted my user the roles/iam.denyReviewer at the Org level and confirmed no Deny Policies exist at Org/Folder/Project levels that would block run.services.invoke or target my user/service.
  6. Organization Policies: Checked effective policies for the project:
    • constraints/run.allowedIngress: Confirmed this is Inactive / Inheriting, likely defaulting to allow-all traffic.
    • Reviewed other Org Policies; none seemed likely to block authenticated access.
  7. VPC Service Controls: Confirmed via Cloud Console (Security → VPC Service Controls) that the project is not part of any VPC-SC perimeter.
  8. Test Container: Deployed Google’s standard us-docker.pkg.dev/cloudrun/container/hello container to a new private service (test-hello-service), granted run.invoker to my user, and tested via curl with access token → Still resulted in 401 Unauthorized. This rules out issues with my specific application/container.
  9. Service Recreation: Completely deleted my-cloud-run-service using gcloud run services delete and redeployed it (privately); the 401 Unauthorized issue persists on the recreated service.
  10. Identity Token Attempt: Briefly attempted using OIDC Identity Tokens, but ran into separate issues generating them (Invalid account Type error for user, NOT_FOUND / PERMISSION_DENIED issues when trying to set up SA impersonation, possibly due to inconsistent Cloud Build SA state).

Summary: Despite confirming the necessary run.invoker IAM role is present for my user, and ruling out standard blockers like IAP, Deny Policies, Org Policy ingress restrictions, and VPC-SC, authenticated requests using standard gcloud access tokens are consistently failing with 401 Unauthorized at the Cloud Run infrastructure level (logs confirm “request was not authenticated”).

Question: Given this extensive troubleshooting, what other configurations, policies (perhaps less common Org Policies?), or potential environmental factors within Google Cloud could be causing this consistent 401 Unauthorizedresponse for authenticated users who demonstrably possess the run.invoker role? We do not have a paid support plan to open a direct technical case.

Any suggestions for further investigation would be extremely helpful!

Thank you.

2

https://cloud.google.com/run/docs/authenticating/developers#curl - Make sure you are using the identity token and not the access token for Auth token:

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $SERVICE_URL$

Note the “print-identity-token” instead of “print-access-token”

3

Thank you for the feedback, particularly the suggestion to use gcloud auth print-identity-token instead of print-access-token for the curl test, as Cloud Run expects an OIDC Identity Token. I agree this is the correct approach according to the documentation.

Unfortunately, I encountered specific errors trying to generate this token in my environment:

  1. Directly as User: Running gcloud auth print-identity-token --audiences=[SERVICE_URL] resulted in ERROR: Invalid account Type for --audiences. Requires valid service account. My user account seems unable to mint audience-specific ID tokens directly.
  2. Via Impersonation: I attempted to impersonate the default Cloud Build SA (...@cloudbuild.gserviceaccount.com) to generate the token. However, granting my user the required Service Account Token Creator role on that SA failed because the SA resource itself consistently returns a NOT_FOUNDerror (likely due to enforced Org Policies: cloudbuild.disableCreateDefaultServiceAccount and cloudbuild.useComputeServiceAccount). Cycling the Cloud Build API after temporarily disabling these policies did not resolve the SA NOT_FOUND issue.

So, while using an Identity Token is the goal, generating one is currently blocked.

The core issue remains: authenticated users (tested multiple accounts) with the run.invoker role consistently receive 401 Unauthorized when accessing private Cloud Run services (including Google’s standard ‘hello’ container) via curlwith an access token. Browser access yields 403 Forbidden. Standard checks (IAM, IAP, Deny Policies, Org Policy Ingress, VPC-SC) haven’t revealed the cause.

Does the inability to manage/find the default Cloud Build SA, or the enforced use of the Compute Engine SA for builds (due to Org Policy), suggest any other potential root causes or troubleshooting avenues for the persistent Cloud Run 401authentication failure?

Thanks again for any insights.

4

Do you need the audience? Does it work without the audience? Agreed, the audience requires an SA, but if you cannot generate an SA could you confirm it works without the audience and then you know you just need a SA? Would confirm the root cause.

5

There are numerous apparent misconceptions to unpack in your post and I do realise some time has come since you posted it so hopefully you will have resolved the issue by now. I am replying because this information may prove useful to someone else in the future as this kind of question keeps popping up now and then in different online communities. So, without further ado…

Firstly, to access a Cloud Run service that requires authentication (i.e. it is not a public service), the HTTP request must include a bearer token in the header. This token has to be of the right kind, though. Cloud Run would accept an ID token [docs, more docs] but not an access token. How you would get an ID token depends on your environment but taking your developer workstation as a starting point, you can, indeed, generate an ID (identity) token with gcloud CLI:

gcloud auth print-identity-token

Note that this is not the same as the access token and the following token would not work to access a Cloud Run service:

gcloud auth print-access-token # this token is not going to work with Cloud Run

For more info about token types, check out these [docs].

Provided that you have the right IAM roles assigned to you, you can test your Cloud Run service with curl [docs]:

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://hello-551927382628.europe-west1.run.app

The docs page I referenced in the previous paragraph lists the necessary permissions for this to work as well as provides some examples of IAM roles that would grant you such permissions. I should not that as of today, the least privileged IAM predefined role that allows you to invoke Cloud Run services is called “Cloud Run Service Invoker“. There is a lot of old docs and blogs out there that say it should be “Cloud Run Invoker” but that role also can invoke Cloud Run Jobs, which is not what we want here.

Thus, provided your service accepts external traffic, has the correct IAM policy set on it, and you are using the right kind of bearer token to send your requests, Cloud Run is very unlikely to reject your request.

Next, when you are logged into Google Cloud, navigating to a Cloud Run service URL in that browser session is not going to help with accessing that service because your browser is not going to include any tokens in that request. This is just how it works. And this is why you were getting error 403 Forbidden. If you would like to authenticate into your Cloud Run service using your Google Account and access the service from your browser, you need to configure Identity-Aware Proxy (IAP) on the Cloud Run service - and this functionality is in preview at the moment. The old school alternative is to put an external load balancer in front of your Cloud Run service and change the service’s settings to only allow access from the VPC and the load balancer backend. Then, the load balancer can handle IAP authentication in the way that you had expected - it could use your Google account to sign in to access your Cloud Run service.

6

Furthermore, you don’t need to use the audiences flag if you are generating an ID token for yourself as a developer. The gcloud auth CLI example in my previous comment works to access Cloud Run services if all the other conditions for access outlined there are met.

Lastly, you can’t impersonate the default Cloud Build service account because it’s not a user-managed account. It belongs to Google. Please see this Note in the docs. This is why you got an error when you attempted to grant yourself a Token Creator role on that Google-managed service account. Just like any other service account, it has an IAM policy attached to it, but since it is not in your project, you don’t have the permissions necessary to modify that service account’s IAM policy and so you can’t impersonate it.

I feel it is worth to mention that best practice for Cloud Build is to use user-managed service accounts for your build jobs [docs]. You don’t have to use the legacy Cloud Build service account or even the Compute Engine “default” service account.

In fact, the best practice for the default Compute Engine service account is to revoke its Editor basic role on your project because it has very broad privileges on the entire project. An interesting consequence of revoking the basic role from the default Compute Engine service account is that it then loses its value so one might as well disable it and start creating and configuring a least-privileged user-managed service account for every workload that they run. For more information, check out the [service accounts best practices doc].

I hope this helps!