Transactions in spanner are converted into a single changestream record. And it looks like https://cloud.google.com/dataflow/docs/guides/templates/provided/cloud-spanner-change-streams-to-pubsub the template (looking at the source code) converts these 1:1 into a pubsub message.
But since spanner transactions are limited by mutation count mostly they can easily exceed the 10MB pubsub limit.
We recently experienced a slew of such errors when making some backfill updates on a spanner table using the parititioned dml api.
textPayload: “Invalid arguments publishing to projects/prod-423420/topics/data-catalog-cdc-topic: The value for request_size is too large. You passed 10311312 in the request, but the maximum value is 10000000..”
Any recommended course of actions for how to solve this? This seems like a significant limitation in the template in that it’s not capable of propagating all changes into the pub/sub destination.