How to schedule BigQuery notebooks with Airflow (Composer) instead of UI scheduler?

Google Cloud Data Analytics
1

Hi everyone,

I’m looking for a way to schedule BigQuery notebooks without relying solely on the built-in UI scheduling tool.

My goal is to orchestrate these notebooks with Cloud Composer (Airflow), so that I can link their execution directly with Dataform transformation tasks.

So far, the only workaround I’ve come up with is:

  • Deploy the notebook to BigQuery + GCS in my YAML pipeline

  • Trigger execution from Airflow by pulling it from GCS and running it on airflow Engine

This approach works but feels far from ideal.

:backhand_index_pointing_right: Is there a recommended or cleaner way to schedule BigQuery notebooks with Airflow/Composer, or integrate them more directly into pipelines with Dataform?

Thanks in advance!

2

Hi Thomas_Lienard,

You might find this related post with the recommended solution helpful. In this answer from @tania, the approach is that these notebooks are executed within Airflow using PythonOperator, which allows them to be monitored, retried, and orchestrated independently.

Additionally, @aga shared an answer in this post that can be also helpful for your case.