need help in figuring out how to make the Vertex AI DAG wait for the task to complete?

[anonymous]

Hi team,

i am trying to create the big query table in Vertex AI but the current implementation makes it difficult for the DAG to wait for the Big query table to be created and hence the DAG fails ultimately.

can you tell me the way on how to wait for the resource creation before moving forward with the next step in DAG?

[lsolatorio]

Hi @HiteshSantwani ,

There are a couple of reasons why your DAG initializes even before the completion of its dependencies:

• The downstream tasks might have been set to wait for the wrong/ incorrect task to finish
• There might still be issues with the DAG trigger wait_for_completion, if ever you are using it

You can try exploring other triggers such as wait_for_downstream or on_success. Make sure to also review your dependencies and set the right triggers.

Related StackOverflow discussions:

• Task in same airflow DAG starts before previous task is committed
• Airflow DAG trigger wait_for_completion not working as expected

Usable resources:

• DAG | Control Flow
• DAG | Control Flow | Trigger Rules
• Customizing DAG Scheduling with Timetables

I hope these links can be of help.