GCP Batch Jobs - due to inadequate quotas Quota: NVIDIA_T4_GPUS, limit: 1, usage: 0, wanted: 2

Google Cloud Compute Infrastructure
, ,
1

hello,

i try to train my model more fast.

i do it with “allocationPolicy”: {
“instances”: [{
“policy”: {“accelerators”: [{“count”: “1”,“type”: “nvidia-tesla-t4”}],“machineType”: “n1-standard-4”,“provisioningModel”: “STANDARD”},“installGpuDrivers”: true}],
“location”: {“allowedLocations”: [“regions/us-east1”]}

and it is fine. i have NVIDIA_T4_GPUS Quota=1 (region : us-east1) and i asked about increasing this quota to 4, but i received inreased quotas “GPUs (all regions)” =4 (i tried more than 3 times to increase NVIDIA_T4_GPUS but received every time increasing “GPUs (all regions)”).

when i set for creation batch\job “accelerators”: [{“count”: “2”,“type”: “nvidia-tesla-t4”}] i received: “Quota checking process decided to delay scheduling for the job job-xxx due to inadequate quotas [Quota: NVIDIA_T4_GPUS, limit: 1, usage: 0, wanted: 2.], next schedule time”.

may be some one was in same situation, and could help me with it ?

2

You are right, “GPUs (all regions)” quota would limit your job but it is not enough - it will allow you to have at max 4 gpus in all regions, each region has its own quota that will be checked too. You’d need to debug the quota requesting process. Steps should be somewhere along the lines of:

Verify Quota Details:

  • Go to the Google Cloud Console.

  • Navigate to IAM & Admin > Quotas.

  • Filter by the service: Compute Engine API.

  • Search for the specific quota name: NVIDIA_T4_GPUS.

  • Check the limit for this quota specifically in the us-east1 region. This is likely still at 1.

  • Also, examine the GPUs (all regions) quota to understand what was increased.

When requesting a quota increase, you need to be very precise. Ensure you are selecting:

  • Service: Compute Engine API

  • Metric: NVIDIA_T4_GPUS (or similar name, verify the exact name in the Quotas page)

  • Location/Region: us-east1

  • Requested Limit: 4

    If your input allows free text add a clear justification in the request, explaining that you need to run Batch jobs requiring multiple T4 GPUs in us-east1 and that previous increases affected the wrong quota.

Preferably make screenshots along the process so if it fails we can debug here further on.