Cloud billing export to BigQuery: Modifying the dataset

Google Cloud Data Analytics
1

Hi! everyone.

I need some help in understanding how the currently configured Cloud Billing export to BigQuery gets affected in case one changes the value of dataset set up for the export (i.e. without disabling the export rather just editing the settings and using another dataset value)?

Does the cost table within the new dataset be automatically backfilled with historical billing data i.e. data prior to the dataset change?

Thanks for the help!

2

Hi @Harsimran14 ,

Welcome to Google Cloud Community!

When you change the dataset for Cloud Billing export, historical billing data won’t be automatically transferred to the new dataset. Only data from the current month and the previous month will be exported to the new dataset. For more information on exporting data to BigQuery, check this documentation.

Here are the proposed solutions:

  • It’s best to store the data in separate tables using a UNION query in BigQuery if you want to combine data from your old configurations and new billing datasets.
  • Compare overlapping partitions between your initial billing export table and new billing tables by using the _PARTITIONTIME column. Once identified, you can delete the overlapping partitions and then perform the final transfer to ensure continuity without duplication.

Note: Before proceeding, create backup tables (copies of your data stored in separate tables) from both your old and new datasets to ensure your data is protected before making any changes. This safeguards your information in case something goes wrong during the process.

I hope the above information is helpful.

1 Like