Infra for reporting

Google Cloud Compute Infrastructure
1

I want to extract data to my project from dataset in same organization.

My plan was that use ‘Dataflow’ for Batch and pipeline job and ETL in BigQuery. However, HQ recommended me to create 2 instances, one for ETL/Batch and Cloud SQL.

My plan options are:

  1. BigQuery (provided from HQ) → Dataflow-> BigQuery (ETL) → Tableau Online

  2. BigQuery (provided from HQ) → Batch Instance-> Cloud SQL (ETL) ↔ BigQuery (my project) → Tableau Online

Should I use Cloud SQL for Database? I planned to use BigQuery for ETL (Inc. Datawarehouse), but HQ recommended me to create Cloud SQL for database and it would be cheaper than just doing ETL in BigQuery.

Which option is more price optimized one, the size of data it not big at this stage.

1 Like