Vertex AI - Slow Batch Predictions

Google Cloud Build with AI Custom ML & MLOps
,
1

Hi,

I’ve been running a Vertex AI Tabular batch prediction job for about 500k rows (50MB BQ table) for nearly 5 hours now, and I can’t see any reference to how it’s performing anywhere. Is there an estimate for how long this should take? Or where I should look for progress?

2

Hi Liam,

You might be able to check results in BigQuery or Cloud Storage [1] to estimate the progress on it. If this does not meet your demand, you might consider filing a feature request per instructions at [2].

[1] https://cloud.google.com/vertex-ai/docs/predictions/batch-predictions#retrieve_batch_prediction_results
[2] https://cloud.google.com/support/docs/issue-trackers

3

Hello,

It’s not possible to check progress by checking the GCS output location. The output files only get uploaded after the job ends.