Setup ETL Data pipeline for batch workload to load BigQuery views into another BigQuery Table

Google Cloud Serverless Applications
,
1

Good Morning

I would like to build data pipeline to load data from one GCP project BQ views into to another GCP project BQ DB. and Automate( requirement is load data on daily basis at regular internal)

Source: Big Query View from one GCP Project

Sink: Big Query Table of another GCP Project

What is best tool to achieve with out of the box serverless services ?

Cloud Dataflow, Cloud Composer,GCS cloud storage

2

I’m not very familiar with BigQuery but as a first step I’d
recommend looking into whether they have any built-in functionality for
this.

If not: Cloud Run jobs is an easy way to run a script on a schedule:
https://cloud.google.com/run/docs/quickstarts/jobs/create-execute

Dataflow and Cloud Composer will work for this too.

Karolina Netolicka
Product Manager, Serverless
knet@google.com