It seems like you’re encountering a quota limit issue with your Vertex AI workflow, specifically related to the number of instances being used by the Dataflow job named ‘dataflow-tabular-stats-and-example-02071145-m2k3-harness’. The error message indicates that the quota limit ‘QUOTA_FOR_INSTANCES’ has been exceeded, with a limit of 24 instances.
First, confirm whether the quota limit for instances (QUOTA_FOR_INSTANCES) in your Google Cloud project has indeed been exceeded. You can do this by navigating to the Google Cloud Console, selecting the project associated with your Vertex AI workflow, and checking the quota limits for Dataflow instances.
If you find that the quota limit has been exceeded, you can request a quota increase from Google Cloud Support. Navigate to the “IAM & Admin” section in the Google Cloud Console, select “Quotas”, then find the quota for Dataflow instances and request an increase.
Ensure that other resources in your project are not consuming too many resources, which could be impacting the available quota for Dataflow instances. This includes other running workflows or services that might be using Compute Engine instances.