There’s currently no plan to support replication from a replica. Can you explain why you want to avoid using CDC on the replica?
thanks @etaim .
No we actually want to make use of the replica to be a source for DataStream.
I think that this is more of a conceptual problem of data replication and does not have to do anything with Datastream or Cloud SQL for Postgres technologies.
You can replicate data in 3 different ways:
a) Single-leader (or leader-follower, master-slave)
b) Multi-leader
c) Leaderless
I assume that you are using single-leader replication in Cloud SQL which implies that followers (replicas) can be used for read-only operations. I would say (but I’m not sure about it) that Datastream needs to write to a source database about acknowledging what has already been successfully replicated, since it is considered as a subscriber in a relationship with a source database.
ENUMs are now supported.
https://cloud.google.com/datastream/docs/release-notes#June_30_2023
Hi @etaim , Now we can create a replication using Datastream and postgres DB which is currently real-time sync/replication to GBQ. We wanted to do the cdc on the same database as the below link for datastream and dataflow analytics which is supporting MySQL/Oracle DBs.
https://cloud.google.com/datastream/docs/implementing-datastream-dataflow-analytics
Can we do this on the postgres DB or is there any chance this will be enabled in near future for analytics on Postgres DBs to capture all the events to GBQ and not only replication sync ?
Your assistance will be greatly appreciated.
We’re planning to support “append-only” mode by EOY, which will capture all events in BigQuery (without applying any MERGE).
There is currently no ETA for updating the Dataflow template to support PostgreSQL, but it is something we are looking into.
I’m happy to share that support for PostgreSQL ARRAYs is now available:
https://cloud.google.com/datastream/docs/release-notes#October_20_2023
We will be building a new application where we need to stream change events from cloudSQL PostgreSQL to a pubsub topic. We checked but Google Datastream currently doesn’t have this support.
Hi! Does Datastream support composite types when replicating a PostgreSQL source to BigQuery? The known limitations don’t explicitly mention it, it just says they’re not supported in arrays. Thanks!