I connected my Cloud PostgreSQL database to BigQuery through DataStream. When I make inserts into my database, the record is added to my BigQuery Tables correctly, but when I delete from my PostgreSQL server the record still shows in my Bigquery tables. The data is not refreshing and the “datastream_metadata.is_deleted” column in Bigquery shows false for rows that I know for a fact I’ve deleted, or truncated from my table.
How do I get DataStream to Recognize DELETE’s in my Postgresql server? At what point in the process is this failing?
Hi @smstreb97 ,
There are a few reasons why the deletion of PostgreSQL data is not reflecting in your BigQuery:
- There might be some incorrect DataStream configuration. The connection profiles for the PostgreSQL and the BigQuery destination should be correct
- The BigQuery destination might not be in a ready state to receive the data and be sure that the table you are trying to mirror has enough space
- The data deletion might not have been committed in PostgreSQL. You can check and review the PostgreSQL logs to see if the deletion actually happened
- The BigQuery sync might not have run yet. The sync happened periodically in the background so make sure that you deleted the data from the PostgreSQL database prior to the sync process
- A problem with the BigQuery sync configuration is a possibility, make sure that the BigQuery sync configuration is correct
After reviewing all of these things and you are still having trouble, you can contact Google Cloud Datastream support for help.
