From some weeks, we successfully created some Vertex AI Search Indexes in the console UI and with Terraform in several projects, some creations lasted around 30 mins, some even only 3 or 4 mins.
From yesterday 27/11, all our Indexes creation are never ending, staying with the ‘Creating’ status for hours (even after more than 4 hours for one of them).
Example of a test in the console UI:
- region: europe-west3
- algorithm type: TreeAH algorithm
- dimensions: 768
- Approximate neighbors count: 150
- update method: Stream
- shard size: Small
- distance measure type: Dot product distance
- feature norm type: None
- fraction leaf nodes to search: 0.05
- lean node embedding count: 1000
Example of a test with Terraform:
resource “google_vertex_ai_index” “vertexai_index” {
project = module.target_project.project_id
region = “europe-west3”
display_name = “vertexai-index”
description = “vertexai-index”
metadata {
contents_delta_uri = “gs://${google_storage_bucket.vertexai_index_bucket.name}/contents”
config {
dimensions = 768
approximate_neighbors_count = 150
shard_size = “SHARD_SIZE_SMALL”
distance_measure_type = “DOT_PRODUCT_DISTANCE”
algorithm_config {
tree_ah_config {
leaf_node_embedding_count = 500
leaf_nodes_to_search_percent = 7
}
}
}
}
index_update_method = “STREAM_UPDATE”
depends_on = [
google_storage_bucket_object.data
]
}
What we have tested / checked so far:
- Tried creating 5 or 6 indexes with europe-west3 or us-central1
- confirmed in the documentation that creation can take up to 1 hour:
https://cloud.google.com/vertex-ai/docs/vector-search/create-manage-index
“Your new index appears in your list of indexes once it’s ready. Note: Build time can take up to an hour to complete.”
However we are far from 1 hour here, even after 4 / 5 hours still not created - checked permissions, the terraform service account and the UI users have been granted Owner for troubleshooting reasons, still the issue was there
- quotas on the newly created project seem ok (everything green)
Vertex AI API / Matching Engine concurrent index creation operations. per region: 1/ 5
Vertex AI API / Matching Engine Indexes per region: 1 / 15 - everything green for quotas at organization level
- No error found in Cloud Logging, only the index never created, status keeps showing Creating after some hours (more than 4 hours for an index creation today and still not created)
- gcloud ai indexes list or describe --region= : return “0 items” and "NOT_FOUND: Index
projects/<project id>/locations/europe-west3/indexes/<index id>is not found
We do not know what else to check:
- There is no list method for “gcloud ai operations”, we do not know how to get the operations id and try to get some clarity with the describe command
(https://cloud.google.com/sdk/gcloud/reference/ai/operations) - not sure if using another dimensions number or approximate neighbors count or any other parameter would change something
Would someone have a clue ?
