Hi community,
I have a question related to managing data quality rules as code with Terraform which is described here - https://cloud.google.com/dataplex/docs/manage-data-quality-rules-as-code#setting_up_your_github_repository
Nevertheless, it works, my question is how to manage creation of several scans-since based on example and article we set only one table with rules for it. How can we define rules for several tables and deploy them?
Hi @Arstem ,
Welcome to Google Cloud Community!
While Dataplex offers advanced use cases, you cannot define data quality rules for multiple tables within a single task as of the moment.
As @ms4446 mentioned about limitation:
Dataplex’s Data Quality rules focus on single tables, with sql_expression designed for specific table contexts. While subqueries are supported, their use is limited to operations within the primary table of the Data Quality scan.
Here’s a workaround that you can do, create one YAML file per table containing its rules, then use Terraform to deploy a separate google_dataplex_task for each YAML file, ensuring one task per table.
I hope the above information is helpful.
Thank you for your response. But I believe there may have been a misunderstanding. My concern is about managing data quality rules as code using the Terraform-provided solution.
In the current implementation, it processes only one YAML file with rules for a specific table at a time. But what if I want to manage rules for multiple tables - say 10 YAML files? Ideally, I would like to run the solution and have it create 10 corresponding scans in Dataplex automatically.
Hi @Arstem
I have de similar quetion, did you find any domcument to help you?