How to trigger a data quality in dataplex after a dag succes

Google Cloud Data Analytics
1

Hi team

I have a question regarding data quality rules. All my data quality rules are in YAML files, which are interpreted by Terraform for deployment. So far, so good. However, now the need is that before executing a data quality job, it should verify a DAG (a DAG that transforms a table, for example, customer_table), and then perform the data quality check.
How should I proceed in this scenario?

2

Hi @jvilchezv ,

You can achieve this by setting up your DAG (e.g., in Cloud Composer / Airflow) to trigger the Dataplex data quality job as a task after the main transformation step.

3

Hi @jvilchezv ,

Welcome to Google Cloud Community!

You can easily trigger actions like cleanup or tag application based on Dataplex reports using EventArc and Cloud Audit Logs. When a data quality run detects an issue, it generates a log record in Cloud Logging. EventArc can then be configured to monitor these logs and trigger actions when a specific event is detected.

To trigger a Dataplex Data Quality scan after a successful Dataflow job, you can use a Cloud Function:

  1. Create a Dataflow job: Make sure your Dataflow job is set up and running correctly.
  2. Set up a Cloud Function: This function will be triggered upon the Dataflow job’s completion.
  3. Configure the Cloud Function trigger: Use EventArc to trigger the Cloud Function when the Dataflow job succeeds.
  4. Add code to trigger the Data Quality scan: Include code in the Cloud Function to start the Dataplex Data Quality scan. You can use the Dataplex API or a custom script for this purpose.
  5. Deploy the Cloud Function: Deploy the function so it can listen for the Dataflow job completion event and trigger the Data Quality scan.

You can also use Terraform and Cloud Build to manage different release versions. A Git repository can be used to define data quality rules in YAML files, which are then parsed by a Python script and provisioned as data quality scans using Terraform. This approach allows you to version control your Data Quality rules and reuse them across multiple columns and tables.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

4

Hi @dawnberdan

Thanks for that information. I’ll check it.

1 Like