How do I determine if there is an issue with Cloud Secrets Manager in the AU Region ?
My cloud run services are having issues communicating with secrets manager.
I am now getting
Total timeout of API google.cloud.secretmanager.v1.SecretManagerService exceeded 60000 milliseconds
This service has been running for months, deployed a simple update, now will not communicate with Secrets Manager.
This is in australia-southeast1
Any direction / help on how to determine the issue or if an outage with GCP would be helpful.
1 Like
Jb83
November 8, 2023, 7:39pm
2
I am also currently having issues with the client libraries accessing secrets from Google Secret Manager (Europe).
We are getting “502 Bad Gateway” from a majority of the requests. I thought this was related to incident https://status.cloud.google.com/incidents/VgryWZ3bxKBSUEBjXaSb#oKGDLmoneB3YgW79P7vm but it says that is resolved. Our issues started in conjuncture to this incident happening and hasn’t stopped. Any ideas?
Jb83
Which version of the Client Libraries are you using? I am using NodeJS ver 3.7.3
“@google-cloud /secret-manager”: “^3.7.3”,
Hmm… going to look at upgrading.
I just created a Cloud Function that reads secrets from Secret Manager. Running NodeJS.
I created this function in US East Coast, EU Belgium, and AU Sydney. The function only fails within the Sydney region with Timeout.
Jb83,
My issue “can” be resolved if I reference the Project Number, not the Project ID.
Might be related to the outage you reference, not sure.
I am having the same issue on the Sydney server.
I’ve been using the project Number* for the last 6 months with no issues, but today the issue appeared for the first time.
Yes, it is all related to the Sydney Cloud Run and Cloud Functions attempting to access Secrets Manager.
I have not found a solution, other than not use Secrets Manager on NEW installs. Apparently existing deployments appear to be working, just new deployments.
I re-published a cloud function that uses it and it worked for 2 days after republishing, then got this error this morning. Hopefully it’s discovered and resolved quickly, it’s not mentioned on their service health page yet.https://status.cloud.google.com/regional/asia
We are having critical issues too. All our services are Sydney based. Cloud run and Cloud Functions cannot resolve secrets.
Having the same issue about 10 minutes ago, we are requesting secret manager from App Engine, it’s in the Sydney region as well
+1 timeouts accessing Secret Manager Service via App Engine in the Sydney region. First occurrance we can see in the logs was at 6:28am ADST.
2 Likes
sbsleb
November 8, 2023, 11:16pm
13
We are having outages on all our production accounts, timeout of 60s can’t access secret manager cloud run/cloud functions, can’t seem to reproduce locally.
1 Like
We can run up our app locally with no issues accessing secret manager. It only happens when deployed to App Engine.
We have seen a few successes now. Maybe 1% of requests making it through
To those affected by this, there are two solutions until Google fixes the issue.
Remove use of Secrets Manager (yeah, not good solution)
Move your Cloud Functions / Cloud Runs / App Engines to another region.
This is so unacceptable. No communication at all from Google.
1 Like
Gotta love status pages…always the last to know
1 Like
Are we able to be compensated for this? How does it work? I’ve got customers screaming at me, and my only solution is to wait. I can’t easily switch regions, or hard code secrets.
You can sign up for support: https://console.cloud.google.com/support
After paying, you can get your incident escalated.
This