Scaling trust: How to build autonomous metadata governance loops using Gemini and Google Cloud Dataplex

The Enterprise Metadata Blindspot

Modern enterprise data platforms host thousands of schemas, tables, and views spanning BigQuery, Spanner, and external data lakes. While Google Cloud Knowledge Catalog (formerly Dataplex) acts as the centralized knowledge hub, enterprise catalogs suffer from severe metadata decay:

  • Lineage Black Holes: Tables created by ad-hoc queries or external pipelines lack registered upstream transformations or downstream consumers.

  • Unclassified Sensitive Data: Columns containing PII, financial information, or secrets lack proper Taxonomy Tag templates and fine-grained column-level IAM policies.

  • Documentation Debt: Mission-critical tables have empty descriptions, undefined business glossary links, and ambiguous abbreviations.

  • Stale & Low-Quality Assets: Tables experience silent schema drift or fail Data Quality (Auto DQ) threshold checks without alerting data consumers.

Traditional rule-based validation scripts fail because metadata governance requires semantic reasoning: understanding context, inferring business definitions, and evaluating risk.

In this article, we will build an Autonomous Metadata Governance Agent using Python, Google Cloud Knowledge catalog, and Gemini on Vertex AI. We will deploy the multi-agent system on Google Cloud Run to continuously audit databases, identify critical metadata gaps, and generate automated Dataplex Tag remediation payloads.


Architecture Overview

The solution leverages an orchestrator agent coordinating three specialized audit sub-agents:

  1. Catalog Extractor: Pulls entry schemas, aspect types, column profiles, and active lineage graph nodes from the Knowledge Catalog API.

  2. Lineage & Quality Agent: Traverses upstream/downstream nodes to detect broken DAGs, orphaned assets, and low data quality scores.

  3. Taxonomy & Security Agent: Analyzes column names and data distributions using Gemini to spot unclassified PII, missing policy tags, and security masking omissions.

  4. Documentation & Schema Drift Agent: Identifies missing business glossaries, blank descriptions, and unmonitored schema mutations.

  5. Remediation Orchestrator: Synthesizes the overall Catalog Health Index (CHI) and pushes remediation tags back into Dataplex.


       [ Scheduled Trigger / Eventarc ]
                     β”‚
                     β–Ό
          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
          β”‚   Orchestrator Agent β”‚ (Cloud Run)
          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                     β”‚  Pulls Entries & Aspects
                     β–Ό
         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
         β”‚ Google Cloud Knowledge catalog  β”‚ (Universal Catalog)
         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                    β”‚
       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
       β”‚            β”‚                        β”‚
       β–Ό            β–Ό                        β–Ό
 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
 β”‚  Lineage  β”‚ β”‚  Taxonomy &   β”‚ β”‚ Documentation & Drift  β”‚
 β”‚ & Quality β”‚ β”‚  Security     β”‚ β”‚ Agent (Gemini 3.8)     β”‚
 β”‚   Agent   β”‚ β”‚  Agent        β”‚ β”‚                        β”‚
 β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
       β”‚               β”‚                     β”‚
       β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                       β–Ό
       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
       β”‚ Synthesized Gap Report &      β”‚
       β”‚ Automated Dataplex Tag Patch  β”‚
       β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜


1. Setup Dependencies

Create a requirements.txt file containing the Google Cloud client libraries, FastAPI, and OpenTelemetry instrumentation:

text


fastapi>=0.110.0
uvicorn>=0.28.0
httpx>=0.27.0
google-cloud-dataplex>=2.0.0
google-cloud-datacatalog>=3.19.0
google-cloud-bigquery>=3.20.0
google-genai>=0.1.1
pydantic>=2.6.0
opentelemetry-api>=1.24.0
opentelemetry-sdk>=1.24.0
opentelemetry-instrumentation-fastapi>=0.45b0


2. Knowledge Catalog & Metadata Retrieval Layer

Create catalog_client.py to fetch entries, column schemas, and existing tag aspects from Dataplex:

python


# catalog_client.py
import os
from google.cloud import dataplex_v1
from google.cloud import datacatalog_v1

class DataplexMetadataClient:
    def __init__(self, project_id: str, location: str = "us-central1"):
        self.project_id = project_id
        self.location = location
        self.datacatalog_client = datacatalog_v1.DataCatalogClient()
        self.dataplex_client = dataplex_v1.DataplexServiceClient()

    def get_table_metadata(self, entry_name: str) -> dict:
        """Fetches table schema, column descriptions, and attached tags."""
        try:
            entry = self.datacatalog_client.get_entry(name=entry_name)
            tags = list(self.datacatalog_client.list_tags(parent=entry.name))
            
            columns = []
            if entry.schema and entry.schema.columns:
                for col in entry.schema.columns:
                    columns.append({
                        "name": col.column,
                        "type": col.type_,
                        "description": col.description or "",
                        "mode": col.mode or "NULLABLE",
                    })

            attached_tags = [
                {
                    "template": t.template_display_name or t.template,
                    "fields": {k: str(v.string_value or v.enum_value) for k, v in t.fields.items()}
                }
                for t in tags
            ]

            return {
                "entry_name": entry.name,
                "display_name": entry.display_name,
                "description": entry.description or "",
                "linked_resource": entry.linked_resource,
                "columns": columns,
                "tags": attached_tags,
            }
        except Exception as e:
            return {"error": f"Failed to retrieve entry {entry_name}: {str(e)}"}


3. Implementing Specialized Governance Sub-Agents

A. Lineage & Upstream Continuity Auditor

Create lineage_agent.py to evaluate whether data assets have valid transformation histories and active downstream consumers:

python


# lineage_agent.py
from google.cloud import datacatalog_lineage_v1

class LineageAuditor:
    def __init__(self, project_id: str, location: str = "us-central1"):
        self.client = datacatalog_lineage_v1.LineageClient()
        self.location = location
        self.project_id = project_id

    def audit_lineage(self, linked_resource: str) -> dict:
        """Verifies if the table has linked lineage processes."""
        parent = f"projects/{self.project_id}/locations/{self.location}"
        request = datacatalog_lineage_v1.SearchLinksRequest(
            parent=parent,
            target=datacatalog_lineage_v1.EntityReference(fully_qualified_name=linked_resource)
        )
        
        links = list(self.client.search_links(request=request))
        
        has_upstream = any(link.target.fully_qualified_name == linked_resource for link in links)
        has_downstream = any(link.source.fully_qualified_name == linked_resource for link in links)
        
        gaps = []
        if not links:
            gaps.append("CRITICAL_LINEAGE_GAP: No upstream or downstream lineage registered.")
        elif not has_upstream:
            gaps.append("ORPHAN_SOURCE_GAP: Table has downstream consumers but no documented source pipeline.")
        elif not has_downstream:
            gaps.append("TERMINAL_SINK_GAP: Table has upstream inputs but no registered downstream consumers.")

        return {
            "link_count": len(links),
            "lineage_gaps": gaps,
            "status": "PASS" if not gaps else "FAIL"
        }


B. Taxonomy, PII & Policy Masking Detective

Create security_agent.py using Gemini 3.8 to evaluate column names, infer sensitive data types, and check against existing Dataplex Policy Tags:

python


# security_agent.py
import json
from google import genai
from google.genai import types

class SecurityTaxonomyAuditor:
    def __init__(self):
        self.client = genai.Client()

    def audit_security_and_pii(self, table_name: str, columns: list[dict], existing_tags: list[dict]) -> dict:
        """Detects unclassified sensitive columns and missing policy tags."""
        schema_context = json.dumps(columns, indent=2)
        tags_context = json.dumps(existing_tags, indent=2)

        prompt = f"""
        You are a Cloud Data Governance and Compliance Expert inspecting Google Cloud Dataplex metadata.
        
        Table: {table_name}
        Columns:
        {schema_context}
        
        Existing Applied Tags:
        {tags_context}
        
        Tasks:
        1. Identify any columns containing PII, Financial Data, Secrets, or Credentials that LACK policy/taxonomy tags.
        2. Identify if column-level data masking policies appear missing for sensitive fields.
        3. Output strict JSON with the following format:
        {{
            "unclassified_pii_columns": ["col1", "col2"],
            "missing_policy_tags": ["High_Security_PII", "Mask_Email"],
            "risk_level": "HIGH | MEDIUM | LOW",
            "justification": "Explanation of risks"
        }}
        """

        response = self.client.models.generate_content(
            model="gemini-2.5-pro",
            contents=prompt,
            config=types.GenerateContentConfig(
                response_mime_type="application/json"
            )
        )
        return json.loads(response.text)


C. Freshness, Quality & Schema Evolution Auditor

Create drift_agent.py to inspect missing business documentation and data freshness indicators:

python


# drift_agent.py
import json
from google import genai
from google.genai import types

class DriftAndQualityAuditor:
    def __init__(self):
        self.client = genai.Client()

    def audit_documentation_and_drift(self, table_metadata: dict) -> dict:
        """Evaluates documentation completeness, empty descriptions, and business glossary alignment."""
        columns = table_metadata.get("columns", [])
        empty_cols = [c["name"] for c in columns if not c.get("description")]
        table_desc = table_metadata.get("description", "")

        prompt = f"""
        Evaluate metadata completeness for database table: {table_metadata.get('display_name')}.
        Table Description: '{table_desc}'
        Columns with empty descriptions: {empty_cols}
        Total columns: {len(columns)}

        Produce recommended business descriptions for missing attributes and identify catalog gaps.
        Respond in JSON:
        {{
            "missing_table_description": {"true" if not table_desc else "false"},
            "missing_column_descriptions_count": {len(empty_cols)},
            "missing_column_list": {json.dumps(empty_cols)},
            "suggested_table_description": "Clear professional description if missing",
            "glossary_terms_needed": ["Term1", "Term2"]
        }}
        """

        response = self.client.models.generate_content(
            model="gemini-2.5-flash",
            contents=prompt,
            config=types.GenerateContentConfig(
                response_mime_type="application/json"
            )
        )
        return json.loads(response.text)


4. Multi-Agent Orchestrator & Remediation Engine

Create orchestrator.py using FastAPI. The orchestrator triggers all sub-agents, aggregates the gaps into a Catalog Health Score, and provides an automated remediation payload ready for datacatalog.create_tag:

python


# orchestrator.py
import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from catalog_client import DataplexMetadataClient
from lineage_agent import LineageAuditor
from security_agent import SecurityTaxonomyAuditor
from drift_agent import DriftAndQualityAuditor

app = FastAPI(title="Dataplex Autonomous Metadata Governance Engine")

PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-gcp-project")
metadata_client = DataplexMetadataClient(project_id=PROJECT_ID)
lineage_auditor = LineageAuditor(project_id=PROJECT_ID)
security_auditor = SecurityTaxonomyAuditor()
drift_auditor = DriftAndQualityAuditor()

class AuditRequest(BaseModel):
    entry_name: str  # Format: projects/{p}/locations/{l}/entryGroups/{g}/entries/{e}

@app.post("/audit-entry")
async def run_metadata_audit(request: AuditRequest):
    # 1. Fetch live Dataplex metadata
    meta = metadata_client.get_table_metadata(request.entry_name)
    if "error" in meta:
        raise HTTPException(status_code=404, detail=meta["error"])

    linked_resource = meta.get("linked_resource", "")

    # 2. Run Sub-Agent Audits
    lineage_results = lineage_auditor.audit_lineage(linked_resource)
    security_results = security_auditor.audit_security_and_pii(
        table_name=meta.get("display_name", ""),
        columns=meta.get("columns", []),
        existing_tags=meta.get("tags", [])
    )
    drift_results = drift_auditor.audit_documentation_and_drift(meta)

    # 3. Calculate Catalog Health Index (0-100)
    penalties = 0
    if lineage_results["lineage_gaps"]: penalties += 30
    if security_results["unclassified_pii_columns"]: penalties += 40
    if drift_results["missing_table_description"]: penalties += 15
    if drift_results["missing_column_descriptions_count"] > 0: penalties += 15

    health_score = max(0, 100 - penalties)

    # 4. Generate Remediation Plan
    return {
        "entry": meta["display_name"],
        "catalog_health_score": health_score,
        "gaps_detected": {
            "lineage": lineage_results["lineage_gaps"],
            "security_and_taxonomy": security_results,
            "documentation_and_glossary": drift_results
        },
        "remediation_action": {
            "apply_dataplex_aspect_tag": {
                "template": "projects/my-project/locations/us-central1/tagTemplates/governance_compliance_aspect",
                "fields": {
                    "audit_status": "FLAGGED_FOR_REMEDIATION",
                    "pii_risk_level": security_results["risk_level"],
                    "suggested_description": drift_results["suggested_table_description"]
                }
            }
        }
    }


5. Containerization & Deployment to Cloud Run

Dockerfile

dockerfile


FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT=8080
CMD uvicorn orchestrator:app --host 0.0.0.0 --port ${PORT}

Deployment Commands

Enable the necessary APIs:

bash


gcloud services enable \
    run.googleapis.com \
    datacatalog.googleapis.com \
    dataplex.googleapis.com \
    datalineage.googleapis.com \
    aiplatform.googleapis.com

Grant Cloud Run’s Service Account access to knowledge catalog and Vertex AI:

bash


PROJECT_ID=$(gcloud config get-value project)
PROJECT_NUM=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
SA_EMAIL="${PROJECT_NUM}-compute@developer.gserviceaccount.com"

# Grant roles for Data Catalog, Dataplex, and Vertex AI
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/datacatalog.viewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/datacatalog.tagEditor"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/datalineage.viewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/aiplatform.user"

Deploy the multi-agent service:

bash


gcloud run deploy dataplex-metadata-agent \
    --source . \
    --region us-central1 \
    --set-env-vars "GCP_PROJECT_ID=${PROJECT_ID}" \
    --no-allow-unauthenticated \
    --quiet


6. Testing & Verifying Catalog Gap Remediation

Retrieve your Cloud Run service URL and trigger an audit against a target BigQuery entry in Dataplex:

bash


SERVICE_URL=$(gcloud run services describe dataplex-metadata-agent --region us-central1 --format="value(status.url)")

ENTRY_NAME="projects/${PROJECT_ID}/locations/us-central1/entryGroups/@bigquery/entries/$(echo -n "bigquery.googleapis.com/projects/${PROJECT_ID}/datasets/analytics_dw/tables/user_transactions" | base64)"

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  -H "Content-Type: application/json" \
  -d "{\"entry_name\": \"${ENTRY_NAME}\"}" \
  "${SERVICE_URL}/audit-entry"

Execution Output Example

JSON


{
  "entry": "user_transactions",
  "catalog_health_score": 30,
  "gaps_detected": {
    "lineage": [
      "TERMINAL_SINK_GAP: Table has upstream inputs but no registered downstream consumers."
    ],
    "security_and_taxonomy": {
      "unclassified_pii_columns": ["customer_email", "billing_address", "ip_hash"],
      "missing_policy_tags": ["Confidential_PII_Taxonomy/Email", "Confidential_PII_Taxonomy/Address"],
      "risk_level": "HIGH",
      "justification": "Direct PII stored in plain text with no column-level masking policy or IAM restriction tags attached."
    },
    "documentation_and_glossary": {
      "missing_table_description": true,
      "missing_column_descriptions_count": 8,
      "suggested_table_description": "Stores user payment transaction histories, merchant identifiers, and checkout session metadata for financial reconciliation.",
      "glossary_terms_needed": ["Customer_Account", "Gross_Merchandise_Value"]
    }
  },
  "remediation_action": {
    "apply_dataplex_aspect_tag": {
      "template": "projects/my-project/locations/us-central1/tagTemplates/governance_compliance_aspect",
      "fields": {
        "audit_status": "FLAGGED_FOR_REMEDIATION",
        "pii_risk_level": "HIGH"
      }
    }
  }
}


Conclusion

Static metadata catalogs decay the moment new pipelines launch. By combining Google Cloud Knowledge Catalog with an Agentic AI governance loop, organizations can:

  1. Eliminate Blindspots: Proactively detect unclassified PII, missing policy tags, and severed lineage edges before compliance audits flag them.

  2. Automate Documentation: Eliminate documentation debt by generating accurate column descriptions and business glossary mappings with Gemini 3.8.

  3. Close the Loop: Empower agents to directly generate Dataplex Tag templates and remediation entries, turning governance from a reactive chore into an autonomous continuous control plane.

Happy cataloging!


Large data platforms can have thousands of schemas, tables, and views spread across BigQuery, Spanner, and external data lakes. Having a catalog does not automatically mean that the metadata inside it is still useful.

Over time, metadata starts getting out of date.

A table might have no proper lineage even though it is being used by several pipelines. A column can contain customer information or financial data without having the right policy tags. Important tables may have empty descriptions, while other tables slowly develop schema changes that nobody documents.

This is what metadata decay looks like.

Google Cloud Dataplex, now part of the Google Cloud knowledge catalog services, gives teams a central place to manage this information. The problem is that someone still needs to keep checking whether the metadata is complete and correct.

A simple script can check whether a description exists. It can check whether a tag is present. But it cannot easily understand that a column called customer_email probably contains personal information, or that a table with no documented upstream source is a governance problem.

That is where an AI-based approach can help.

The idea here is to build a small metadata governance system using Python, Google Cloud Dataplex, Gemini on Vertex AI, and Cloud Run. The service reads metadata from the catalog, sends different parts of the audit to specialized agents, combines their results, and produces a report showing what needs attention.

The system is split into a few smaller parts instead of putting everything into one large function.

The first part reads the catalog and collects the table schema, columns, descriptions, tags, and linked resources.

The lineage agent then checks whether the table has upstream and downstream lineage.

The security agent looks at the columns and existing tags and uses Gemini to identify things that may contain PII, financial information, secrets, or credentials.

Another agent checks the documentation. It looks for missing table descriptions, empty column descriptions, and possible glossary terms.

Finally, the orchestrator takes all of these results and produces one health score along with a remediation payload.

The basic flow looks like this:

Scheduled Trigger / Eventarc
            |
            v
     Orchestrator Agent
        Cloud Run
            |
            v
    Google Cloud Dataplex
            |
      +-----+------+----------------+
      |            |                |
      v            v                v
   Lineage     Taxonomy &       Documentation
   & Quality     Security        & Drift Agent
      |            |                |
      +------------+----------------+
                   |
                   v
        Gap Report + Remediation

The dependencies are fairly small. The service needs the Google Cloud clients, FastAPI for the API endpoint, the Gemini SDK, Pydantic, and OpenTelemetry.

fastapi>=0.110.0
uvicorn>=0.28.0
httpx>=0.27.0
google-cloud-dataplex>=2.0.0
google-cloud-datacatalog>=3.19.0
google-cloud-bigquery>=3.20.0
google-genai>=0.1.1
pydantic>=2.6.0
opentelemetry-api>=1.24.0
opentelemetry-sdk>=1.24.0
opentelemetry-instrumentation-fastapi>=0.45b0

The first thing the application needs is a way to read metadata from the catalog.

For that, catalog_client.py creates the Dataplex and Data Catalog clients and exposes a method that receives an entry name and returns the information needed by the other agents.

from google.cloud import dataplex_v1
from google.cloud import datacatalog_v1

class DataplexMetadataClient:
    def __init__(self, project_id: str, location: str = "us-central1"):
        self.project_id = project_id
        self.location = location

        self.datacatalog_client = datacatalog_v1.DataCatalogClient()
        self.dataplex_client = dataplex_v1.DataplexServiceClient()

    def get_table_metadata(self, entry_name: str) -> dict:
        try:
            entry = self.datacatalog_client.get_entry(name=entry_name)
            tags = list(
                self.datacatalog_client.list_tags(parent=entry.name)
            )

            columns = []

            if entry.schema and entry.schema.columns:
                for col in entry.schema.columns:
                    columns.append({
                        "name": col.column,
                        "type": col.type_,
                        "description": col.description or "",
                        "mode": col.mode or "NULLABLE"
                    })

            attached_tags = [
                {
                    "template": t.template_display_name or t.template,
                    "fields": {
                        k: str(v.string_value or v.enum_value)
                        for k, v in t.fields.items()
                    }
                }
                for t in tags
            ]

            return {
                "entry_name": entry.name,
                "display_name": entry.display_name,
                "description": entry.description or "",
                "linked_resource": entry.linked_resource,
                "columns": columns,
                "tags": attached_tags
            }

        except Exception as e:
            return {
                "error": f"Failed to retrieve entry {entry_name}: {str(e)}"
            }

This part is intentionally simple. The other agents should not have to know how to talk to Dataplex. They just receive a normal Python dictionary containing the metadata they need.

The lineage check works differently because it needs to query the lineage service.

For each table, the agent searches for registered lineage links. It then checks whether there are upstream and downstream connections.

from google.cloud import datacatalog_lineage_v1

class LineageAuditor:
    def __init__(self, project_id: str, location: str = "us-central1"):
        self.client = datacatalog_lineage_v1.LineageClient()
        self.location = location
        self.project_id = project_id

    def audit_lineage(self, linked_resource: str) -> dict:
        parent = f"projects/{self.project_id}/locations/{self.location}"

        request = datacatalog_lineage_v1.SearchLinksRequest(
            parent=parent,
            target=datacatalog_lineage_v1.EntityReference(
                fully_qualified_name=linked_resource
            )
        )

        links = list(self.client.search_links(request=request))

        has_upstream = any(
            link.target.fully_qualified_name == linked_resource
            for link in links
        )

        has_downstream = any(
            link.source.fully_qualified_name == linked_resource
            for link in links
        )

        gaps = []

        if not links:
            gaps.append(
                "CRITICAL_LINEAGE_GAP: "
                "No upstream or downstream lineage registered."
            )
        elif not has_upstream:
            gaps.append(
                "ORPHAN_SOURCE_GAP: "
                "Table has downstream consumers but no documented source pipeline."
            )
        elif not has_downstream:
            gaps.append(
                "TERMINAL_SINK_GAP: "
                "Table has upstream inputs but no registered downstream consumers."
            )

        return {
            "link_count": len(links),
            "lineage_gaps": gaps,
            "status": "PASS" if not gaps else "FAIL"
        }

The useful part here is that the agent does not just return True or False. It explains what kind of problem it found.

The security check is where Gemini becomes useful.

A schema by itself does not explicitly tell us that customer_email is personal information. The name gives us a strong clue, but the application still needs to make a judgment based on the available context.

The security agent sends the column information and existing tags to Gemini and asks it to identify potentially sensitive fields.

import json
from google import genai
from google.genai import types

class SecurityTaxonomyAuditor:
    def __init__(self):
        self.client = genai.Client()

    def audit_security_and_pii(
        self,
        table_name: str,
        columns: list[dict],
        existing_tags: list[dict]
    ) -> dict:

        schema_context = json.dumps(columns, indent=2)
        tags_context = json.dumps(existing_tags, indent=2)

        prompt = f"""
        You are a Cloud Data Governance and Compliance Expert
        inspecting Google Cloud Dataplex metadata.

        Table: {table_name}

        Columns:
        {schema_context}

        Existing Applied Tags:
        {tags_context}

        Tasks:

        1. Identify columns containing PII, Financial Data,
           Secrets, or Credentials that lack policy or taxonomy tags.

        2. Identify whether column-level masking policies
           appear to be missing for sensitive fields.

        3. Return strict JSON using this structure:

        {{
            "unclassified_pii_columns": ["col1", "col2"],
            "missing_policy_tags": ["High_Security_PII"],
            "risk_level": "HIGH | MEDIUM | LOW",
            "justification": "Explanation of risks"
        }}
        """

        response = self.client.models.generate_content(
            model="gemini-2.5-pro",
            contents=prompt,
            config=types.GenerateContentConfig(
                response_mime_type="application/json"
            )
        )

        return json.loads(response.text)

The important part is the output format. The application is not asking Gemini for a paragraph that a developer then has to parse manually. It asks for JSON with known fields, so the orchestrator can use the result directly.

The documentation and drift agent follows a similar idea, but its job is different.

It first checks which columns have no descriptions. It also checks whether the table itself has a description. Gemini can then suggest a description and possible business glossary terms.

class DriftAndQualityAuditor:
    def __init__(self):
        self.client = genai.Client()

    def audit_documentation_and_drift(
        self,
        table_metadata: dict
    ) -> dict:

        columns = table_metadata.get("columns", [])

        empty_cols = [
            c["name"]
            for c in columns
            if not c.get("description")
        ]

        table_desc = table_metadata.get("description", "")

        prompt = f"""
        Evaluate metadata completeness for database table:
        {table_metadata.get('display_name')}.

        Table Description:
        '{table_desc}'

        Columns with empty descriptions:
        {empty_cols}

        Total columns:
        {len(columns)}

        Produce recommended business descriptions for missing
        attributes and identify catalog gaps.

        Respond in JSON:

        {{
            "missing_table_description": true,
            "missing_column_descriptions_count": 0,
            "missing_column_list": [],
            "suggested_table_description": "...",
            "glossary_terms_needed": ["Term1", "Term2"]
        }}
        """

        response = self.client.models.generate_content(
            model="gemini-2.5-flash",
            contents=prompt,
            config=types.GenerateContentConfig(
                response_mime_type="application/json"
            )
        )

        return json.loads(response.text)

At this point we have three separate results.

The orchestrator is responsible for putting them together.

The API is built with FastAPI. When /audit-entry receives an entry name, it first gets the metadata. Then it sends that metadata to the lineage, security, and documentation agents.

After that, it calculates a simple Catalog Health Score.

@app.post("/audit-entry")
async def run_metadata_audit(request: AuditRequest):

    meta = metadata_client.get_table_metadata(
        request.entry_name
    )

    if "error" in meta:
        raise HTTPException(
            status_code=404,
            detail=meta["error"]
        )

    linked_resource = meta.get("linked_resource", "")

    lineage_results = lineage_auditor.audit_lineage(
        linked_resource
    )

    security_results = (
        security_auditor.audit_security_and_pii(
            table_name=meta.get("display_name", ""),
            columns=meta.get("columns", []),
            existing_tags=meta.get("tags", [])
        )
    )

    drift_results = (
        drift_auditor.audit_documentation_and_drift(meta)
    )

    penalties = 0

    if lineage_results["lineage_gaps"]:
        penalties += 30

    if security_results["unclassified_pii_columns"]:
        penalties += 40

    if drift_results["missing_table_description"]:
        penalties += 15

    if drift_results["missing_column_descriptions_count"] > 0:
        penalties += 15

    health_score = max(0, 100 - penalties)

    return {
        "entry": meta["display_name"],
        "catalog_health_score": health_score,

        "gaps_detected": {
            "lineage": lineage_results["lineage_gaps"],
            "security_and_taxonomy": security_results,
            "documentation_and_glossary": drift_results
        },

        "remediation_action": {
            "apply_dataplex_aspect_tag": {
                "template": "...",
                "fields": {
                    "audit_status": "FLAGGED_FOR_REMEDIATION",
                    "pii_risk_level": security_results["risk_level"],
                    "suggested_description":
                        drift_results[
                            "suggested_table_description"
                        ]
                }
            }
        }
    }

The score itself is deliberately simple in this example.

A lineage problem removes 30 points. Unclassified PII removes 40. Missing table documentation removes 15, and missing column descriptions remove another 15.

This gives the team one number to look at, while the actual response still contains the details behind that number.

For example, a table could come back with a score of 30 because it has a lineage problem, unclassified PII, and missing documentation.

The response might look like this:

{
  "entry": "user_transactions",
  "catalog_health_score": 30,

  "gaps_detected": {
    "lineage": [
      "TERMINAL_SINK_GAP: Table has upstream inputs but no registered downstream consumers."
    ],

    "security_and_taxonomy": {
      "unclassified_pii_columns": [
        "customer_email",
        "billing_address",
        "ip_hash"
      ],
      "missing_policy_tags": [
        "Confidential_PII_Taxonomy/Email",
        "Confidential_PII_Taxonomy/Address"
      ],
      "risk_level": "HIGH"
    },

    "documentation_and_glossary": {
      "missing_table_description": true,
      "missing_column_descriptions_count": 8,
      "suggested_table_description":
        "Stores user payment transaction histories, merchant identifiers, and checkout session metadata for financial reconciliation.",
      "glossary_terms_needed": [
        "Customer_Account",
        "Gross_Merchandise_Value"
      ]
    }
  }
}

Now the service can be packaged as a small container and deployed to Cloud Run.

The Dockerfile does not need much:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

ENV PORT=8080

CMD uvicorn orchestrator:app \
    --host 0.0.0.0 \
    --port ${PORT}

Before deployment, the required Google Cloud APIs need to be enabled:

gcloud services enable \
    run.googleapis.com \
    datacatalog.googleapis.com \
    dataplex.googleapis.com \
    datalineage.googleapis.com \
    aiplatform.googleapis.com

The Cloud Run service account also needs access to the services the application is using.

For example:

PROJECT_ID=$(gcloud config get-value project)

PROJECT_NUM=$(
    gcloud projects describe $PROJECT_ID \
    --format="value(projectNumber)"
)

SA_EMAIL="${PROJECT_NUM}-compute@developer.gserviceaccount.com"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/datacatalog.viewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/datacatalog.tagEditor"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/datalineage.viewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/aiplatform.user"

Then the service can be deployed:

gcloud run deploy dataplex-metadata-agent \
    --source . \
    --region us-central1 \
    --set-env-vars "GCP_PROJECT_ID=${PROJECT_ID}" \
    --no-allow-unauthenticated \
    --quiet

Once Cloud Run is running, the API can be called against a Dataplex entry.

SERVICE_URL=$(
    gcloud run services describe dataplex-metadata-agent \
    --region us-central1 \
    --format="value(status.url)"
)

ENTRY_NAME="projects/${PROJECT_ID}/locations/us-central1/entryGroups/@bigquery/entries/$(echo -n "bigquery.googleapis.com/projects/${PROJECT_ID}/datasets/analytics_dw/tables/user_transactions" | base64)"

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
    -H "Content-Type: application/json" \
    -d "{\"entry_name\": \"${ENTRY_NAME}\"}" \
    "${SERVICE_URL}/audit-entry"

The result gives you more than just a pass or fail.