Issue:
Around 5-10% of messages from push subscriber is failing to be pushed to the cloud run service and is accumulating the dead letter queue. The issue is happening even for 1-2 TPS of workload. Also apart from DLQ messages accumulating, the instance count of the service is increasing from 1 to 15 instances.
The pubsub subscriber has 5 retries starting from 10 secs to 600 secs with exponential back-off.
We tried generating load by hitting the cloud run service directly (without any pubsub) using plain HTTPS at 20 TPS. It’s still able to handle the load with just 1 instance.
No observed error message.
We adopted a pubsub+cloud-run architechture for more scalability but its behaving worse. Can anyone please help us understand the issue?