[CRITICAL] Cloud Secret Manager global endpoint not redirecting to working region

Google Cloud Serverless Applications
1

I have Google Cloud Secrets which are replicated in multiple Europe regions. The europe-west-9 region is currently down and my services located in europe-west-1 region cannot access my secrets ! Even if those secrets are replicated between multiple regions.

Any tricks to make my programs running in Cloud Function (europe-west-1) access my secrets?

Seems that the design of the replication of GCP Secrets is a failure because the global endpoint does not redirect to a working region.

PS : I note that the issue with Secret Manager global endpoint is not recognized publicly. A simple call from my laptop shows that the system is not working:
$ gcloud secrets list
ERROR: (gcloud.secrets.list) HttpError accessing <https://secretmanager.googleapis.com/v1/projects/xxxxx/secrets?alt=json&pageSize=100>: response: <{‘vary’: ‘Origin, X-Origin, Referer’, ‘content-type’: ‘application/json; charset=UTF-8’, ‘content-encoding’: ‘gzip’, ‘date’: ‘Wed, 26 Apr 2023 12:16:02 GMT’, ‘server’: ‘ESF’, ‘cache-control’: ‘private’, ‘x-xss-protection’: ‘0’, ‘x-frame-options’: ‘SAMEORIGIN’, ‘x-content-type-options’: ‘nosniff’, ‘alt-svc’: ‘h3=“:443”; ma=2592000,h3-29=“:443”; ma=2592000’, ‘transfer-encoding’: ‘chunked’, ‘status’: 503}>, content <{
“error”: {
“code”: 503,
“message”: “The service is currently unavailable.”,
“status”: “UNAVAILABLE”
}
}

This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.

3 Likes
2

Facing a similar issue with requests returning HTTP 500 or 503 or hanging:

$ gcloud version
Google Cloud SDK 428.0.0
alpha 2023.04.25
beta 2023.04.25
bq 2.0.91
bundled-python3-unix 3.9.16
core 2023.04.25
gcloud-crc32c 1.0.0
gsutil 5.23
$

$ curl -V
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
$

$ curl -sSkv https://secretmanager.googleapis.com/v1/projects/mkdev/secrets --header "authorization: Bearer XXXXX" --header "content-type: application/json"
* Trying 142.250.178.138:443...
* TCP_NODELAY set
* Connected to secretmanager.googleapis.com (142.250.178.138) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=upload.video.google.com
* start date: Apr 3 08:24:24 2023 GMT
* expire date: Jun 26 08:24:23 2023 GMT
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55719bc948f0)
> GET /v1/projects/mkdev/secrets HTTP/2
> Host: secretmanager.googleapis.com
> user-agent: curl/7.68.0
> accept: */*
> authorization: Bearer XXXXX
> content-type: application/json
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 500
< vary: X-Origin
< vary: Referer
< vary: Origin,Accept-Encoding
< content-type: application/json; charset=UTF-8
< date: Wed, 26 Apr 2023 14:31:29 GMT
< server: ESF
< cache-control: private
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< accept-ranges: none
<
{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
* Connection #0 to host secretmanager.googleapis.com left intact
$
$ curl -sSkv https://secretmanager.googleapis.com/v1/projects/mkdev/secrets --header "authorization: Bearer XXXXX" --header "content-type: application/json"
* Trying 172.217.18.202:443...
* TCP_NODELAY set
* Connected to secretmanager.googleapis.com (172.217.18.202) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=upload.video.google.com
* start date: Apr 3 08:24:24 2023 GMT
* expire date: Jun 26 08:24:23 2023 GMT
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b59bf298f0)
> GET /v1/projects/mkdev/secrets HTTP/2
> Host: secretmanager.googleapis.com
> user-agent: curl/7.68.0
> accept: */*
> authorization: Bearer XXXXX
> content-type: application/json
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 500
< vary: X-Origin
< vary: Referer
< vary: Origin,Accept-Encoding
< content-type: application/json; charset=UTF-8
< date: Wed, 26 Apr 2023 14:35:00 GMT
< server: ESF
< cache-control: private
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< accept-ranges: none
<
{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
* Connection #0 to host secretmanager.googleapis.com left intact
$

$ nslookup secretmanager.googleapis.com | sort -u

Address: 10.18.0.50#53
Address: 142.250.178.138
Address: 142.250.179.106
Address: 142.250.179.74
Address: 142.250.201.170
Address: 142.250.75.234
Address: 172.217.18.202
Address: 172.217.20.170
Address: 172.217.20.202
Address: 216.58.213.74
Address: 216.58.214.170
Address: 216.58.214.74
Address: 2a00:1450:4007:807::200a
Address: 2a00:1450:4007:80b::200a
Address: 2a00:1450:4007:80d::200a
Address: 2a00:1450:4007:80e::200a
Name: secretmanager.googleapis.com
Non-authoritative answer:
Server: 10.18.0.50
$

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux MKD4411 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$

Requests work on another non-WSL Ubuntu 20.04.5 VM with same curl version and Google Cloud SDK 408.0.1. Not clear what the differenciating factor is.

3

Also getting 500s when using the gcloud CLI.

4

Same issue on may side, we cannot operate without any alternative command to reach our secrets and deploy:

gcloud secrets versions access "latest" --secret "my-secret-in-current-gcp-project"

Generates the following error whatever the europe-west-1 GCP project and the requested secret:

ERROR: (gcloud.secrets.versions.access) HttpError accessing <https://secretmanager.googleapis.com/v1/projects/my-gcp-project/secrets/my-secret-in-current-gcp-project/versions/latest:access?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'content-encoding': 'gzip', 'date': 'Wed, 26 Apr 2023 21:23:21 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000', 'transfer-encoding': 'chunked', 'status': 500}>, content <{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}
>
This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.
5

seems to be solved now