Google Cloud Storage Transfer Service - triggers?

Google Cloud Compute Infrastructure
,
1

Hi,

Simple (I hope!) question about the google cloud storage transfer service. I’m potentially going to copy a million or so audio assets from S3 to Google. Two questions:

  1. When each object is transferred to Google storage, do any triggers get honoured? I have a trigger on the target bucket and I need that to go off on ‘object finalise’ for each incoming object.

  2. Is there any way to limit the transfer rate? (I suspect not, reading the docs)

Thanks,

A.

2

I tested a transfer job between two Cloud Storage buckets, and Object Finalize triggers do process for each object uploaded. However, given that Cloud Storage events are delivered with Pub/Sub, the delivery guarantees of Pub/Sub notifications will apply, in case it affects your use case.

As for your second question, there is no direct way to change the rate of transfer from a transfer job. If you’d like to ensure the transfer happens as quickly as possible, please follow the best practices for transfer jobs.

1 Like
3

Thanks - that’s really helpful. In our situation we actually want to slow it down a bit as one part of our processing pipeline is capacity-limited. So we’ll probably go with a code approach so we can control that more easily.

Thanks again for your help,

Antony.