Appending Table in BigQuery

Google Cloud Data Analytics
,
1

I am wanting to create a Table in BQ that has all of our direct mail data. We have 1.3 million rows of mail data right now. All of our mail data is in csv’s. I combined all of our mail data into one csv file, but it was too big to locally upload the data. So I had to upload the csv to Cloud Storage. I now have created a table in BQ that has all of this mail data. The problem is I have to update this mail data each week, because we send 50,000+ direct mail pieces a week. I need to figure out a way to update/append this mail table in BQ each week with our new Mail Data. What are my options?

ALSO: I can easily just spend 5 minutes to update this table each week, there is no reason for me to automate this.

2

To update or append your mail table in BigQuery with new mail data each week, you can directly upload the new CSV file containing the updated data to BigQuery.

3

If I’m understanding your issue correctly, you are going to love external tables.

Basically you can define a table in BigQuery to read csv files in a GCS bucket. Then any time you add a csv file to the bucket, the table would reflect the new rows from that csv in the table.

Just be really careful to create your csv files in the exact same format.