Google Cloud Run certificate expired - then new provisioning never completes

Google Cloud Serverless Applications
1

All of the sudden I’ve found out that the certificate for our app expired, basically Google Cloud Run failed to renew it on time. In a desperate attempt a deleted the the custom domain name mapping and then created a new mapping hopping I will force it to create a new certificate. Unfortunately the certificate provisioning from the mapping seems to never complete(see the message below).

Waiting for certificate provisioning. You must configure your DNS records for certificate issuance to begin.

I can see several requests in the logs that seem to end-up with http 302 redirects. If I make the same requests(s) using cURL (e.g. http://example.com/.well-known/acme-challenge/6iZfcMFcs5XshVnclYj834_fIrNBvawpVN_Z2GRYpfHVkwgf8HyiJ9gUud3eHHw_ ) I get http 200 with the challenge response not 302 as it appears in the logs

`

{
“insertId”: “651144c400011e19725c2d9e”,
“httpRequest”: {
“requestMethod”: “GET”,
“requestUrl”: "http://example/.well-known/acme-challenge/6iZfcMFcs5XshVnclYj834_fIrNBvawpVN_Z2GRYpfHVkwgf8HyiJ9gUud3eHHw_ ",
“requestSize”: “577”,
“status”: 302,
“userAgent”: “Google-Certificates-Bridge”,
“remoteIp”: “172.69.70.160”,
“serverIp”: “216.239.38.21”,
“latency”: “0s”,
“protocol”: “HTTP/1.1”
},
“resource”: {
“type”: “cloud_run_revision”,
“labels”: {
“project_id”: “db-smsv-uk”,
“service_name”: “coz-ro”,
“configuration_name”: “coz-ro”,
“revision_name”: “coz-ro-01322-wp6”,
“location”: “europe-west1”
}
},
“timestamp”: “2023-09-25T08:28:52.071951Z”,
“severity”: “INFO”,
“logName”: “projects/db-smsv-uk/logs/run.googleapis.com%2Frequests”,
“trace”: “projects/db-smsv-uk/traces/d0897b8c1a2f3971d989a49875b3b22a”,
“receiveTimestamp”: “2023-09-25T08:28:52.077545978Z”,
“spanId”: “13580890275068216318”,
“traceSampled”: true
}

`

CURL request

``

curl -v "http://example.com/.well-known/acme-challenge/6iZfcMFcs5XshVnclYj834_fIrNBvawpVN_Z2GRYpfHVkwgf8HyiJ9gUud3eHHw_ "

``

``

  • Trying 104.21.23.56:80…
  • Connected to example.comm (104.21.23.56) port 80 (#0)

GET /.well-known/acme-challenge/6iZfcMFcs5XshVnclYj834_fIrNBvawpVN_Z2GRYpfHVkwgf8HyiJ9gUud3eHHw_ HTTP/1.1
Host: example.com
User-Agent: curl/7.84.0
Accept: /

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Date: Mon, 25 Sep 2023 08:51:14 GMT
    < Content-Type: text/plain; charset=UTF-8
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < Cache-Control: private
    < X-XSS-Protection: 0
    < X-Frame-Options: SAMEORIGIN
    < CF-Cache-Status: DYNAMIC
    < Report-To: {“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report/v3?s=zLrrmfhVtDZ9eOXRZ1W2CZUu7AOqJ8dFUcduim8VRBgdhWvKZreC%2BmCYKyZjiFjK2Xq7x1g5t1Dr6kmp7095b8vBFQhAGFN63a1aoNzIoBmIhVdegjB4Kxk%3D”}],“group”:“cf-nel”,“max_age”:604800}
    < NEL: {“success_fraction”:0,“report_to”:“cf-nel”,“max_age”:604800}
    < Server: cloudflare
    < CF-RAY: 80c20630aef329fd-ORD
    < alt-svc: h3=“:443”; ma=86400
    <
  • Connection #0 to host example.com left intact
    6iZfcMFcs5XshVnclYj834_fIrNBvawpVN_Z2GRYpfHVkwgf8HyiJ9gUud3eHHw_.EC6LW0mQfd3F7Tfc8A7OpH8uuC2dyRUg3KSQqchUufo%

``

2

Hi @themihai1 ,

Welcome to Google Cloud Community!

You can try a few of the following:

  • Make sure your DNS records are correctly configured. The DNS records for your domain must be configured to point to Google Cloud Run’s load balancer. You can check the DNS records for your domain in your domain registrar’s control panel.
  • Verify your domain ownership. Google Cloud Run needs to verify that you own the domain that you’re trying to map to your service. You can do this by adding a TXT record to your domain’s DNS records.
  • Wait at least 24 hours. Certificate provisioning can take up to 24 hours to complete. If you’ve made sure that your DNS records are correct and you’ve verified your domain ownership, but the certificate provisioning is still stuck, try waiting for 24 hours and then checking back.
  • Contact Google Cloud support. If you’ve tried all of the above and the certificate provisioning is still stuck, you can contact Google Cloud support for assistance.

Regarding the 302 redirects you’re seeing in the logs:

It’s possible that the 302 redirects are being caused by a caching issue. If you’re using a content delivery network (CDN), try clearing the CDN cache. You can also try using a different browser or device to access your service.

If you’re still seeing 302 redirects after clearing the CDN cache and trying a different browser or device, you can contact Google Cloud support for assistance.

Here are some additional troubleshooting tips:

  • Make sure that you’re using the correct DNS records for your domain. You can find the correct DNS records in the Google Cloud console.
  • Make sure that your domain is registered with a valid domain registrar.
  • Make sure that your SSL/TLS certificate is valid and not expired.
3

I’ve fixed the redirect & certificate issue by disabling the proxy feature of the DNS provider(Cloudflare).

1 Like
4

@themihai1 did you keep your proxy disabled ? Seems like a very insecure workaround ?