Immediate "429 Resource has been exhausted" on Speech-to-Text v2 (Chirp 3) in US region, but works fine in EU

Google Cloud Build with AI AI APIs
1

Hello,

I am experiencing a persistent “429 Resource has been exhausted” error on my very first request when calling the Google Cloud Speech-to-Text v2 API with the “chirp_3” model in the “us” region.

Here are the details of my setup and what I’ve verified:

  1. Project & Billing:

    • The project has a fully active and open billing account linked.
    • The Speech-to-Text API is enabled.

  2. Quota Check:

    • I checked the quotas via “gcloud beta quotas info list”. The limits for the “us” region are non-zero:
      • UsBatchRequestsPerMinutePerProject: 128
      • UsSyncRequestsPerMinutePerProject: 246
      • UsOperationRequestsPerMinutePerProject: 150
      • UsResourceRequestsPerMinutePerProject: 100
      • AudiosecondsRequestsPerDayPerProject (Global): 1,728,000 seconds

  3. The Issue (US Region):

    • Using the client option endpoint “us-speech.googleapis.com” and recognizer “projects/{PROJECT}/locations/us/recognizers/_”, any request immediately returns:
      “StatusCode.RESOURCE_EXHAUSTED - Resource has been exhausted (e.g. check quota).”
    • This occurs on the very first API call after a server restart. No actual quota has been consumed.
    • Interestingly, even simple utility calls like “get_operation” or “cancel_operation” to the “us” endpoint immediately fail with the same “429 ResourceExhausted” error.

  4. Comparison (EU Region Works):

    • Using the exact same credentials and target project, if I switch the endpoint to the “eu” region (“eu-speech.googleapis.com” and “projects/{PROJECT}/locations/eu/recognizers/_”), the API works perfectly.
    • The “BatchRecognizeRequest” starts successfully, and “get_operation” returns “done = False” and eventually finishes transcription without any 429 throttle issues.

Why would the “us” region immediately block all requests with a 429 error on a billing-active project when the configured quotas are positive and unused? Is there an undocumented regional restriction or trust-level throttling for the “us” region Chirp 3 model?

Any advice would be highly appreciated. Thank you!

2

did you find a solution?

7

Update

I found a workaround that appears to have resolved the issue on my side.

Initially, I followed the suggestion from this thread and switched the processing region from US to EU. Although that reduced the frequency of the 429 errors, I don’t believe it addressed the root cause.

After investigating further, I discovered that the main issue was the behavior of the google-api-core Python library.

My application polls the Long Running Operation only once every 45 seconds, but the SDK applies its own automatic retry policy. Whenever google.longrunning.Operations.GetOperation receives a 429 (Resource Exhausted) response, the library immediately performs multiple internal retries, causing what should be a single polling request to become several requests in just a few milliseconds.

Those internal retries quickly exhaust the quota for google.longrunning.Operations.GetOperation.

Instead of relying on the SDK defaults, I implemented my own retry strategy with controlled Exponential Backoff and explicit polling intervals. The application now controls exactly when a new status request is sent, preventing bursts of requests generated by the SDK.

A simplified version of the approach looks like this:

from google.api_core import retry as api_core_retry
from google.api_core import exceptions

custom_retry = api_core_retry.Retry(
    initial=15.0,
    maximum=60.0,
    multiplier=2.0,
    deadline=3600.0,
    predicate=api_core_retry.if_exception_type(
        exceptions.TooManyRequests,
        exceptions.ResourceExhausted,
        exceptions.InternalServerError,
        exceptions.ServiceUnavailable,
    ),
)

operation = client.batch_recognize(request=req, retry=custom_retry)

poll_interval = 45

while not operation.done(retry=custom_retry):
    time.sleep(poll_interval)

After implementing this change, I have not received any more 429 errors, and the quota consumption for google.longrunning.Operations.GetOperation has dropped dramatically. I’ve been able to process multiple transcription requests successfully without exhausting the operation quota.

It seems that controlling and limiting the SDK polling behavior is much more effective than relying on the default retry policy. Hopefully this helps anyone else running into the same issue.