Next-Gen Data Pipelines: Airflow 3 arrives on Google Cloud Composer

Google Cloud Community Articles
, , ,
1

In the dynamic world of data engineering, staying ahead of the curve is paramount. Teams are constantly seeking tools that not only manage the complexities of data pipelines but also empower them to innovate faster and more reliably. At Google Cloud, we’re committed to being at the forefront of this evolution.

Today, we’re thrilled to announce that Google is the first hyperscaler to provide selected customers with access to Apache Airflow 3, integrated into our fully managed Cloud Composer 3 service. This is a significant step forward, allowing data teams to explore the next generation of workflow orchestration within a robust and familiar Google Cloud environment. While this is an early look, it’s a testament to our dedication to bringing cutting-edge capabilities to our users swiftly.

What features does Airflow 3 bring or plan in the near future?

Airflow 3, brings a suite of powerful capabilities designed to significantly enhance how data workflows are built, managed, and monitored:

  • DAG Versioning: A crucial feature for auditability and stable pipeline evolution. Teams can now track changes, manage different versions of their DAGs, and roll back with confidence.
  • Scheduler-Managed Backfills: Reprocessing historical data is now simpler and more robust. Backfills are managed directly within the scheduler and can be triggered and monitored via the UI or API.
  • Task Execution API & SDK: This opens up new possibilities in the future for tasks to be written in multiple languages, not just Python. It also enables tasks to run in isolated environments, offering greater flexibility and decoupling task logic.
  • Modern React-based UI: Experience a faster, more responsive interface. With features like dark mode and improved navigation, operational oversight becomes more intuitive and efficient.
  • Event-Driven Scheduling & Data Assets (Planned): This planned capability aims to make pipelines more reactive. The goal is for DAGs to be triggered by external events or changes in data assets, enabling near real-time processing.
  • Edge Executor: Optimize task execution by running tasks closer to data sources or on specialized hardware through distributed execution in remote environments.
  • Split CLI (airflow and airflowctl): A clearer command-line experience. This distinction between local development commands (airflow) and operational/remote commands (airflowctl) improves usability and supports a more service-oriented architecture.

Why experience Airflow 3 on Cloud Composer?

Experiencing Airflow 3 on Cloud Composer isn’t just about new features; it’s about leveraging them within a fully managed, enterprise-grade environment. This means you can significantly reduce your operational overhead. Instead of wrestling with infrastructure, your team can focus on what truly matters: building innovative data pipelines.

With Cloud Composer, you benefit from:

  • Robust GCP Security: Leverage Google Cloud’s comprehensive security measures to protect your critical data workflows.
  • Seamless Ecosystem Integration: Connect effortlessly with the broader Google Cloud ecosystem, from BigQuery to Vertex AI.
  • Stringent compliance standards: Adhere to critical compliance standards, like FedRAMP, with confidence.

The new capabilities in Airflow 3 make it an even more potent tool for orchestrating a multitude of use cases. We see incredible potential, particularly in orchestrating end-to-end MLOps and Generative AI workflows, areas where robust and flexible orchestration is key.

Looking ahead and getting started

We’re excited for selected customers to begin exploring the power of Airflow 3 on Cloud Composer 3 (1). We are diligently working towards making full General Availability (GA) support of Airflow 3 on Cloud Composer 3 a reality, and look forward to sharing more updates soon.

This initiative underscores our commitment at Google Cloud and within the Cloud Composer team to be at the forefront of innovation, providing you with the tools you need to tackle your most ambitious data challenges.

Stay tuned for more updates, and we encourage eligible customers to start exploring these powerful new capabilities!

  1. Please note that as this is an early preview of Airflow 3 on Cloud Composer 3, not all features may be immediately available.
8 Likes
2

As someone deeply involved in building and scaling Airflow based data platforms, this announcement is incredibly exciting. Features like DAG versioning, scheduler-managed backfills, and the modern React UI address real orchestration pain points. The roadmap ahead especially event-driven scheduling and language-agnostic task execution signals a bold step toward more flexible and intelligent pipelines.

Thanks** @Piotr_Wieczorek **and the Cloud Composer team for driving this forward. Truly looking forward to exploring Airflow 3 in depth and seeing how it shapes the future of data engineering. :rocket: .

1 Like
3

Totally feel you on the Airflow 3 hype, it’s been a real upgrade for most data teams. The new DAG versioning and smoother backfill setup address many of those annoying issues. We used to battle during reruns and patching broken tasks.

I’ve definitely been through those late nights fighting weird job failures and untangling messy DAGs. So yeah, Airflow 3 really feels like a breath of fresh air.

Some of the stuff I’m loving:

  • That DAG versioning is super helpful when you need to roll back or see what broke
  • The split CLI just makes it easier to handle dev and prod separately
  • And the new UI, finally, feels smooth and less heavy for daily stuff

That said, not everyone wants to deal with all the infra and configs. If you’re more into getting things done fast without diving into code, I’ve had a really good time using Windor.ai.

It’s a no-code ETL/ELT connector, syncs your data immediately, and gets you insights without any scripting effort.

4

Thanks, do you know when it will be available from the console?

5

Looking forward to be able to start using Airflow 3 on Cloud Composer!

Is there any information available about the timeline when it is expected that Airflow 3 will become GA for Cloud Composer?

7

Any chance this will be GA soon?
We want to start an initiative to migrate some of our Workloads to Airflow 3 with Cloud Composer

8

Just another request for timeline for GA Composer Airflow3 - have a project I’d love to start with on Airflow3!

1 Like
9

It’s End of August and there is no Airflow 3 support in sight :man_shrugging:
(though it states it in the cloud composer page as preview, it doesn’t show up in the UI)

1 Like
10

Almost 3 months later and it is in fact not available

11

Hi Everyone,

Thanks for the great discussion. We hear your frustration that this isn’t available yet—frankly, we’re just as eager to get this into your hands.

Our #1 priority for General Availability (GA) is ensuring rock-solid stability for your most critical pipelines. We’re currently validating that in a private preview with a small group of customers to make sure we get it right.

The next step is the Public Preview, which is coming soon. This will be your chance to get hands-on for testing and feedback.

I will post an update directly in this thread the moment it’s live.

-Piotr

7 Likes
12

HEllo Piotr,

WE are also eager to try composer based on airflow 3.

WE will contact our account manager to be added to the private preview.

1 Like
13

Hi Piotr,

Thank you for the update last month and for the focus on ensuring rock-solid stability for the GA release. We understand that Airflow 3.0 was a significant update with architectural changes that naturally require thorough validation for a managed service like Cloud Composer. :white_check_mark:

However, it has been about six months since the 3.0 release, and with Airflow 3.1 now available, the desire to adopt the new version is growing. The recent Airflow Summit really highlighted the impressive new features we’re eager to leverage. :smiling_face_with_sunglasses:

This delay is becoming a concern, especially regarding Composer’s ability to keep pace with Airflow’s release cycle. It’s hard not to notice that some competitors have been offering a managed Airflow 3 since its introduction. :thinking:

With that in mind, could you please share a more concrete timeline for when we can expect General Availability for Airflow 3 on Cloud Composer? An updated ETA would be beneficial for our planning. :spiral_calendar:

Thanks for your help and for keeping us informed. :grinning_face:

5 Likes