High response times on some KMS decrypt method calls

Google Cloud Compute Infrastructure
1

Hello, Google Cloud Community,

We’ve been experiencing high latency on google.cloud.kms.v1.KeyManagementService.Decrypt since last week that have caused some of the request we receive to timeout.

Teban_1-1750345185613.png
Teban_1-1750345185613.png2795×1591 159 KB

(We have seen those peaks since last week, It started about last Friday)

Teban_0-1750348059367.png
Teban_0-1750348059367.png1280×336 121 KB

(Method works, but it takes almost 1 min to answer)

Teban_3-1750346186619.png
Teban_3-1750346186619.png1581×164 28.5 KB

(Request is timing out because we don’t expect that response time is that high)

Currently the keyring for those keys in located on global and the services that use it (cloud run) running on us-central1. It’s a node service that uses GC KMS Node.js Client.

It’s deployed on global since we might need to move service to another region if needed. As it’s a latency issue we thought to create a keyring to us-central1, but could be a potential issue later if we need to move our service. Besides it has been working flawlessly last years until now so we thought it could be a issue with the KMS API, but we didn’t find any issue reported on service health about this.

Do you have any idea how to get through this?

Regards,

2

Hi Teban,

Welcome to Google Cloud Community!

An incident affecting multiple Google Cloud products, including Cloud Key Management Service (KMS) and Cloud Run, occurred from June 12 to June 13, 2025.

As a workaround, you can create a new keyring in us-central1, generate a key within it, and temporarily update your service to use this key for decryption. Testing with a regional keyring can help determine if the global location is contributing to the latency. If this resolves the issue, consider migrating your keys accordingly.

If latency persists beyond the incident period, please contact Google Cloud Support for further investigation.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.