Need a way to stop sending duplicate records into GCP Big Query with Stream Storage API

Google Cloud Data Analytics
1

Hi Team,
we want to insert the data into GCP bigquery with Storage API, we are able to achieve to insert data through batches.
But we don’t want to send duplicate records into big query. also, we don’t have requirement to use unique key record in data. Please let us know how we can achieve.

Thank you.

2

This article looks like it may be a good starting point.

Exactly once delivery in BigQuery’s Storage Write API

3

Hi Kolban, Thanks for your reply.
In the above link it says to use kafka stream and offset. but we want in batches to send the data, please help us on with batches storage api.