Eventarc, how to use a custom (encrypted) topic to trigger cloud function with dataflow events

Google Cloud Data Analytics
1

Good day, I am trying to setup a google cloud function with eventarc as trigger (dataflow event), everything is working but the problem is that the topic that eventarc creates when deploying the cloud function is not custom managed encrypted and this violates the policy, I was wondering if there is a way to specify what topic should the eventarc trigger use, here is the command I use (works but I do not see the option to specify the topic):


gcloud functions deploy eventarc-function-test \
--gen2 \
--region=us-east4 \
--runtime=python312 \
--source=gs://bucket_name/main.zip \
--entry-point=hello_http \
--ingress-settings=internal-only \
--max-instances=1 \
--min-instances=1 \
--concurrency=10 \
--memory=256MB \
--cpu=1 \
--timeout=10s \
--set-env-vars=^~^LOG_EXECUTION_ID=true~CSVHEADERS="value1,value2,value3" \
--build-service-account=projects/project_id/serviceAccounts/project_number-compute@developer.gserviceaccount.com \
--run-service-account=project_number-compute@developer.gserviceaccount.com \
--service-account=project_number-compute@developer.gserviceaccount.com \
--kms-key=projects/project_id/locations/us-east4/keyRings/keyring/cryptoKeys/key \
--docker-repository=us-east4-docker.pkg.dev/project_id/repository \
--trigger-event-filters="type=google.cloud.dataflow.job.v1beta3.statusChanged" \
--trigger-event-filters-path-pattern=job=report-* \
--trigger-location=us-east4 \
--trigger-service-account=project_number-compute@developer.gserviceaccount.com \

I was reading the documentation https://cloud.google.com/sdk/gcloud/reference/functions/deploy#–trigger-topic and though that --trigger-topic would be the way, but then it failed and the reason is that I can not specify the topic unless the event I want to listen to is the google.cloud.pubsub.topic.v1.messagePublished not the dataflow one, so im a little bit stuck, also the documentation says that

You must specify one of the following when deploying a new function:

  • –trigger-topic,
  • –trigger-bucket,
  • –trigger-http,
  • –trigger-event AND --trigger-resource,
  • –trigger-event-filters and optionally --trigger-event-filters-path-pattern.

At most one of these can be specified

Can someone please confirm if it is not possible to encrypt the topic that eventarc uses?

Also I do not think it would be a problem, because even if that topic is google managed encryption, since the eventarc channel for the particular region does have custom managed encryption then the events/messages that it contains would be protected right? Appreciate any help

2

Hi @lawlesspattern ,

Welcome to Google Cloud Community!

You are correct that you can’t directly specify a custom-managed encrypted Pub/Sub topic for Eventarc triggers based on Dataflow events. The event type google.cloud.pubsub.topic.v1.messagePublished only works in –trigger-topic flag.

Alternatively, consider submitting a feature request to specify a custom-managed topic for Dataflow event triggers in Cloud Function with Eventarc. While I can’t provide a timeline for when this will be implemented, I recommend keeping an eye on the tracker and checking the release notes and documents for the latest updates.

I hope the above information is helpful.

1 Like
3

Thanks, yes also the direct support team from Google confirmed this