Total transparency, I am a novice and gemini has been walking through most of these steps. I am trying to stand up this environment on a trial cloud account before purchasing a higher-tier of cloud services, but so far, we cannot get my environment to work.
I am trying to create a simple Cloud Function (we tried both 1st and 2nd gen) in a new project that is triggered by a Cloud Storage file upload. The function consistently fails to start with a “Default STARTUP TCP probe failed” and “Container called exit(1)” error.
This failure only happens when the code imports and initializes any standard Google Cloud client library (like storage.Client() or firestore.Client()).
What I have proven:
-
A “Hello World” function with no Google Cloud libraries deploys and runs successfully.
-
The problem is not IAM, as we have meticulously verified all permissions for both the function’s service account and the Eventarc Service Agent.
-
The problem is not the Cloud Function generation, as it fails in both 1st and 2nd gen environments.
-
The problem is not the project, as the error persists in a brand new, clean project.
-
The problem is not a disabled API or memory allocation.
The function fails during the client library initialization, before any of my own code’s logic can run. I am looking for potential causes related to project/organization policies or other environmental factors that could prevent these libraries from initializing.
Update here after more testing:
I have a Cloud Function triggered by Cloud Storage that is failing with a 403 Forbidden (“not authenticated”) error. This proves the Eventarc Service Agent is missing the Cloud Run Invoker permission.
The core problem: When I go to the IAM panel for the function’s underlying Cloud Run service to add the Cloud Run Invoker role to the Eventarc Service Agent, the UI gives a “No change - principal already exists on the policy” error and will not let me save the change.
The IAM system believes the permission exists, but the logs prove it is not being enforced.
This issue has been reproduced in a brand new project with billing enabled, across both 1st and 2nd generation Cloud Functions. A minimal “Hello World” function also fails with the same 403 error. All necessary APIs are enabled.
This appears to be a backend IAM state contradiction. How can I force this permission to apply or otherwise resolve this issue?
Update: Was able to apply the role to the eventarc SA in the IAM and admin page, but still getting the 403 error.
I have a Cloud Function that fails with a 403 Forbidden (“not authenticated”) error when triggered by Cloud Storage and Eventarc.
The core problem: I have repeatedly granted the Cloud Run Invoker role to the Eventarc Service Agent for the function’s underlying Cloud Run service. The IAM console shows the permission is correctly set. However, the logs consistently show a 403 error, proving the permission is not being enforced at runtime.
This appears to be a backend IAM state contradiction. I have already verified the following:
-
The issue persists in a brand new project with billing enabled.
-
It fails for both 1st and 2nd generation Cloud Functions.
-
A minimal “Hello World” function also fails with the same 403 error.
-
All necessary APIs are enabled.
How can I resolve an issue where the IAM policy in the console is not being enforced?
Tried to change the eventarc’s roles through gcloudCL and failed at that as well. We enabled and disabled the API and it still failed.
The Critical Blocker: When I try to grant that permission, the command fails with a NOT_FOUND error, because the Eventarc Service Agent (service-332722117277@gcp-sa-eventarc.iam.gserviceaccount.com) does not exist in my project.
Lots of things to address here.
In your most recent message you wrote:
Gemini says
The Eventarc service agent is automatically created and granted permissions when you create an Eventarc trigger in Google Cloud. It’s not something you manually create.
Something to check: you cited service-LARGENUMBER as the service agent . Are you certain that number is your project number? (Are you sure you are using the correct email addr for your project?) The way to find your project number:
gcloud projects describe $(gcloud config get-value core/project) \
--format="value(projectNumber)"
But keep in mind that the IAM API in GCP exhibits “eventually consistent” behavior. So if you make a change it may take some time (usually ~minutes not ~hours) for the change to propagate to all affected systems.
BUT, you have provided some inconsistent information. Earlier in the message headed “Update here after more testing”, you said :
So that is a different kind of error. That error implies the Eventarc Service Agent exists and it already has the role you are trying to add. That’s not consistent with your later claim that the Eventarc Service Agent does not exist. Which is it?
SEPARATELY, you may have a misunderstanding about how Eventarc works. When you create a trigger, you must specify a Service Account there. That is a user-managed service account. IT is not the Eventarc Service Agent. THAT service account is the one that must have run.invoker permissions on your cloud function.
Thanks for taking the time to give this detailed response. Appreciate it a ton.
So I applied the role in the console (successfully, supposedly) and then I continue to get the errors in the logs. So for the sake of it, I tried to apply the permissions in CL and that’s when it said NOT_FOUND.
I was trying anything to get this to work, so I applied the eventarc invoker role to the eventarc service agent. My specified service agent that I created does have the invoker role applied, and for the sake of trying to get this to work, I applied the OWNER role to my service agent. Still received the same errors.
Still no success.
{
“textPayload”: “Traceback (most recent call last):\n File “/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py”, line 1473, in wsgi_app\n response = self.full_dispatch_request()”,
“insertId”: “68a76c5a000bdeed92c749ae”,
“resource”: {
“type”: “cloud_run_revision”,
“labels”: {
“revision_name”: “xml-ingestor-00017-mrc”,
“project_id”: “iconik-test-v2”,
“service_name”: “xml-ingestor”,
“configuration_name”: “xml-ingestor”,
“location”: “us-central1”
}
},
“timestamp”: “2025-08-21T18:58:34.777965Z”,
“severity”: “ERROR”,
“labels”: {
“instanceId”: “0069c7a988c2e777cbe8935d6147344844ab5c9a899cbfeb6ef4f1153ce02b240a8f23f2b1f1f39e516afa975c1d6eae28cc648af3f742246ba5fbabae7e7751eb085eabdb35e119a7952b8f2f25”,
“run.googleapis.com/base_image_versions”: “us-docker.pkg.dev/serverless-runtimes/google-22/runtimes/python312:python312_20250809_3_12_11_RC00”
},
“logName”: “projects/iconik-test-v2/logs/run.googleapis.com%2Fstderr”,
“receiveTimestamp”: “2025-08-21T18:58:34.783487997Z”,
“errorGroups”: [
{
“id”: “CNHJ1dfV_qLykgE”
}
]
}
I don’t know what that error indicates, but it looks like it’s only the most recent log message. I would like to get a full stack trace - not sure if you can see that or not.
There should be a stack trace like this one:
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 882, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/execution_id.py", line 106, in wrapper
return view_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/__init__.py", line 188, in view_func
function(event)
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/firebase_functions/firestore_fn.py", line 255, in on_document_updated_wrapped
return _firestore_endpoint_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/firebase_functions/firestore_fn.py", line 176, in _firestore_endpoint_handler
func(database_event)
File "/workspace/main.py", line 35, in send_appointment_notification
updated_data = event.after.data()
^^^^^^^^^^^
AttributeError: 'Event' object has no attribute 'after'
But this one is not your stack trace. Only the start of it is the same. To diagnose your situation, you need all the other lines.
There can be all sorts of reasons why you would see something like this:
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
...
Only the full stack will give you the information you need.