Hi!
I’m trying to complete the codelab for intro to Vertex AI pipelines https://codelabs.developers.google.com/vertex-pipelines-intro but my pipeline jobs are always failing to run containers due to the following error:
com.google.cloud.ai.platform.common.errors.AiPlatformException: code=RESOURCE_EXHAUSTED, message=The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_cpus, cause=null; Failed to create custom job for the task. Task: Project number: 732304752533, Job id: 5833738217119023104, Task id: 8910537889008320512, Task name: product-name, Task state: DRIVER_SUCCEEDED, Execution name: projects/732304752533/locations/europe-west4/metadataStores/default/executions/5380095872214830989; Failed to create external task or refresh its state. Task:Project number: 732304752533, Job id: 5833738217119023104, Task id: 8910537889008320512, Task name: product-name, Task state: DRIVER_SUCCEEDED, Execution name: projects/732304752533/locations/europe-west4/metadataStores/default/executions/5380095872214830989; Failed to handle the pipeline task. Task: Project number: 732304752533, Job id: 5833738217119023104, Task id: 8910537889008320512, Task name: product-name, Task state: DRIVER_SUCCEEDED, Execution name: projects/732304752533/locations/europe-west4/metadataStores/default/executions/5380095872214830989
It seems that I’m exceeding quota for this metric:
aiplatform.googleapis.com/custom_model_training_cpus
But after checking IAM quotas I see that I have a quota of 1 for all CPU types and usage of 0
This seems to be similar to these two issues:
- https://www.googlecloudcommunity.com/gc/AI-ML/Receiving-quota-error-when-trying-to-use-bison-chat-model-in/m-p/597616/highlight/true#M2034
- https://www.googlecloudcommunity.com/gc/AI-ML/Vertex-AI-quota-policy-exceed-when-training-custom-model/m-p/470907#M601
Other stuff I checked:
- I checked my billing account and I see no issues reported there
- I tried running the pipeline in us-central1 and it reports the same errors
Is there any other quota I need to enable for Vertex AI pipelines? Thanks in advance!