Since I work with a Google Cloud Premier Partner, I often help clients migrate their workloads from AWS to GCP.
Recently, I encountered a specific challenge where a client wanted to move to GCP but preferred to use Apache Airflow via Composer. However, they raised concerns about cost, limited control, and expressed a preference for Kubernetes, a tool they’re already familiar with.
To address this, I built a proof of concept (POC) tailored to their requirements, mapping their current AWS setup to equivalent GCP services, ensuring alignment with their operational model.
I wrote a blog post based on this experience (without disclosing any client-specific or NDA-protected information) to share with the community. I believe it could help others facing similar decisions:
Check it out here:
https://medium.com/@rasvihostings/deploying-apache-airflow-on-gke-273e8f977e3d
Now, I’m genuinely curious—Is running Airflow on GKE a better choice?
I’d love to hear your thoughts, both from a business and technical perspective. What are the pros and cons of deploying Airflow on GKE versus using Composer?
At the end of the day, the client will decide what works best for them. As a technical advisor, my role is to guide them with the right insights and your opinions can help shape that advice too
Hi Mohamed_rasvi,
Welcome to Google Cloud Community!
When deciding between Google Cloud Composer and Apache Airflow on GKE, the choice depends on your needs for control, cost, and ease of setup. Cloud Composer is a managed service, while running Airflow on GKE gives you full control over the environment.
Google Cloud Composer (Managed Airflow)
Business Perspective:
-
Pros:
-
Reduced Operational: Google manages the underlying infrastructure (GKE cluster, database, Redis, etc.), upgrades Airflow versions, handles patching, and ensures high availability. This frees up your team to focus solely on developing and managing DAGs, rather than infrastructure.
-
Quick setup: you can provision a Composer environment, making it quick to get started with Airflow without deep Kubernetes expertise.
-
Easy Integration with GCP: Composer works seamlessly with other Google Cloud services like BigQuery, Cloud Storage, and Cloud SQL, making it simpler to manage permissions and data flow.
-
SLA and Support: As a managed service, Composer comes with Google’s enterprise-grade SLAs and support.
-
Predictable Pricing: Pricing is more predictable, based on the environment size, but may not always be cheaper than a self-managed setup.- Cons:
-
Higher Potential Cost: Managed services can be more expensive than running Airflow on your own GKE cluster, especially for steady-state deployments.
-
Limited Control and Customization: You have less granular control over the underlying infrastructure, GKE cluster configuration, and specific Airflow component tuning. While you can adjust some Airflow configurations, deeper optimizations might be restricted.
Technical Perspective:
-
Pros:
-
Simplified Airflow Architecture: Composer handles the orchestration of Airflow components (scheduler, web server, workers, database, message queue) on GKE.
-
Automated Scaling: Composer provides automatic scaling of Airflow workers based on workload, helping manage demands.
-
Built-in Monitoring and Logging: Integrates directly with Cloud Monitoring and Cloud Logging for easy visibility into Airflow’s health and DAG execution.
-
Managed Database and Storage: The Airflow metadata database (Cloud SQL) and DAGs/logs storage (Cloud Storage) are fully managed, reducing maintenance burden.- Cons:
-
Less Flexibility for Advanced Use Cases: If your client has very specific networking requirements, custom Kubernetes configurations, or needs to integrate with non-standard services in a highly customized way, Composer’s managed nature might be a hindrance.
-
Specific Airflow Executor Limitations: Composer might have limitations on which Airflow executors you can use or how you can configure them (e.g., historical limitations with KubernetesExecutor, though this has improved).
Apache Airflow on GKE (Self-Managed)
Business Perspective:
-
Pros:
-
Cost Savings: You can save money by optimizing GKE node types, using spot instances, and fine-tuning cluster sizes, especially for long-running workflows.
-
Maximum Control and Customization: You have full control over the GKE cluster, Kubernetes configurations, and every aspect of your Airflow deployment. This allows for deep optimization and tailored solutions.
-
Familiarity with Kubernetes: This is a huge advantage for your client, as they already have Kubernetes expertise. This reduces the learning curve for their operations team and allows them to leverage existing tooling and processes.
-
Faster Access to Latest Airflow Features: You can deploy new Airflow versions as soon as they are released, allowing you to benefit from the latest features and bug fixes.- Cons:
-
Increased Operational: Your team is responsible for managing the GKE cluster, Airflow components (scheduler, web server, workers, database, message queue), upgrades, patching, security, and scaling. This requires significant Kubernetes and Airflow expertise.
-
Higher Skill Requirements: Your team needs strong Kubernetes, Cloud SQL, and general cloud infrastructure expertise.
Technical Perspective:
-
Pros:
-
Full GKE Features: Use advanced GKE capabilities like custom node pools, GPU support, and complex networking configurations.
-
Choice of Executors: You have complete freedom to choose and configure Airflow executors (e.g., CeleryExecutor, KubernetesExecutor) based on your specific workload needs.
-
Debugging Access: Full access to Kubernetes logs and metrics for deep troubleshooting.
-
Infrastructure as Code (IaC): Deploying Airflow on GKE is highly amenable to IaC tools like Terraform or Pulumi, allowing for reproducible and version-controlled infrastructure.
-
Fine-grained Resource Management: Precisely allocate CPU and memory resources to each Airflow component for optimal performance and cost.- Cons:
-
Complexity of Setup and Maintenance: Setting up a scalable, secure Airflow deployment on GKE requires careful planning and ongoing management (e.g., network setup, load balancing).
-
Security Management: You are responsible for securing the GKE cluster, container images, network access, and IAM roles.
-
Database and Message Queue Management: You’ll need to provision and manage a highly available database (e.g., Cloud SQL or a self-managed PostgreSQL on GKE) and a message queue (e.g., Redis on GKE), which adds complexity.
-
Monitoring and Alerting Configuration: While GKE integrates with Cloud Monitoring, you’ll need to set up comprehensive monitoring and alerting for all Airflow components and the underlying cluster.
If your client values cost-efficiency and full control, and has experience with Kubernetes, Airflow on GKE is likely the better choice.
For reference, refer to the following documentation:
Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.