The Enterprise Metadata Blindspot
Modern enterprise data platforms host thousands of schemas, tables, and views spanning BigQuery, Spanner, and external data lakes. While Google Cloud Knowledge Catalog (formerly Dataplex) acts as the centralized knowledge hub, enterprise catalogs suffer from severe metadata decay:
-
Lineage Black Holes: Tables created by ad-hoc queries or external pipelines lack registered upstream transformations or downstream consumers.
-
Unclassified Sensitive Data: Columns containing PII, financial information, or secrets lack proper Taxonomy Tag templates and fine-grained column-level IAM policies.
-
Documentation Debt: Mission-critical tables have empty descriptions, undefined business glossary links, and ambiguous abbreviations.
-
Stale & Low-Quality Assets: Tables experience silent schema drift or fail Data Quality (Auto DQ) threshold checks without alerting data consumers.
Traditional rule-based validation scripts fail because metadata governance requires semantic reasoning: understanding context, inferring business definitions, and evaluating risk.
In this article, we will build an Autonomous Metadata Governance Agent using Python, Google Cloud Knowledge catalog, and Gemini on Vertex AI. We will deploy the multi-agent system on Google Cloud Run to continuously audit databases, identify critical metadata gaps, and generate automated Dataplex Tag remediation payloads.
Architecture Overview
The solution leverages an orchestrator agent coordinating three specialized audit sub-agents:
-
Catalog Extractor: Pulls entry schemas, aspect types, column profiles, and active lineage graph nodes from the Knowledge Catalog API.
-
Lineage & Quality Agent: Traverses upstream/downstream nodes to detect broken DAGs, orphaned assets, and low data quality scores.
-
Taxonomy & Security Agent: Analyzes column names and data distributions using Gemini to spot unclassified PII, missing policy tags, and security masking omissions.
-
Documentation & Schema Drift Agent: Identifies missing business glossaries, blank descriptions, and unmonitored schema mutations.
-
Remediation Orchestrator: Synthesizes the overall Catalog Health Index (CHI) and pushes remediation tags back into Dataplex.
[ Scheduled Trigger / Eventarc ]
β
βΌ
ββββββββββββββββββββββββ
β Orchestrator Agent β (Cloud Run)
ββββββββββββ¬ββββββββββββ
β Pulls Entries & Aspects
βΌ
ββββββββββββββββββββββββββ
β Google Cloud Knowledge catalog β (Universal Catalog)
ββββββββββββ¬ββββββββββββββ
β
ββββββββββββββΌβββββββββββββββββββββββββ
β β β
βΌ βΌ βΌ
βββββββββββββ βββββββββββββββββ ββββββββββββββββββββββββββ
β Lineage β β Taxonomy & β β Documentation & Drift β
β & Quality β β Security β β Agent (Gemini 3.8) β
β Agent β β Agent β β β
βββββββ¬ββββββ βββββββββ¬ββββββββ βββββββββββββ¬βββββββββββββ
β β β
βββββββββββββββββΌββββββββββββββββββββββ
βΌ
βββββββββββββββββββββββββββββββββ
β Synthesized Gap Report & β
β Automated Dataplex Tag Patch β
βββββββββββββββββββββββββββββββββ
1. Setup Dependencies
Create a requirements.txt file containing the Google Cloud client libraries, FastAPI, and OpenTelemetry instrumentation:
text
fastapi>=0.110.0
uvicorn>=0.28.0
httpx>=0.27.0
google-cloud-dataplex>=2.0.0
google-cloud-datacatalog>=3.19.0
google-cloud-bigquery>=3.20.0
google-genai>=0.1.1
pydantic>=2.6.0
opentelemetry-api>=1.24.0
opentelemetry-sdk>=1.24.0
opentelemetry-instrumentation-fastapi>=0.45b0
2. Knowledge Catalog & Metadata Retrieval Layer
Create catalog_client.py to fetch entries, column schemas, and existing tag aspects from Dataplex:
python
# catalog_client.py
import os
from google.cloud import dataplex_v1
from google.cloud import datacatalog_v1
class DataplexMetadataClient:
def __init__(self, project_id: str, location: str = "us-central1"):
self.project_id = project_id
self.location = location
self.datacatalog_client = datacatalog_v1.DataCatalogClient()
self.dataplex_client = dataplex_v1.DataplexServiceClient()
def get_table_metadata(self, entry_name: str) -> dict:
"""Fetches table schema, column descriptions, and attached tags."""
try:
entry = self.datacatalog_client.get_entry(name=entry_name)
tags = list(self.datacatalog_client.list_tags(parent=entry.name))
columns = []
if entry.schema and entry.schema.columns:
for col in entry.schema.columns:
columns.append({
"name": col.column,
"type": col.type_,
"description": col.description or "",
"mode": col.mode or "NULLABLE",
})
attached_tags = [
{
"template": t.template_display_name or t.template,
"fields": {k: str(v.string_value or v.enum_value) for k, v in t.fields.items()}
}
for t in tags
]
return {
"entry_name": entry.name,
"display_name": entry.display_name,
"description": entry.description or "",
"linked_resource": entry.linked_resource,
"columns": columns,
"tags": attached_tags,
}
except Exception as e:
return {"error": f"Failed to retrieve entry {entry_name}: {str(e)}"}
3. Implementing Specialized Governance Sub-Agents
A. Lineage & Upstream Continuity Auditor
Create lineage_agent.py to evaluate whether data assets have valid transformation histories and active downstream consumers:
python
# lineage_agent.py
from google.cloud import datacatalog_lineage_v1
class LineageAuditor:
def __init__(self, project_id: str, location: str = "us-central1"):
self.client = datacatalog_lineage_v1.LineageClient()
self.location = location
self.project_id = project_id
def audit_lineage(self, linked_resource: str) -> dict:
"""Verifies if the table has linked lineage processes."""
parent = f"projects/{self.project_id}/locations/{self.location}"
request = datacatalog_lineage_v1.SearchLinksRequest(
parent=parent,
target=datacatalog_lineage_v1.EntityReference(fully_qualified_name=linked_resource)
)
links = list(self.client.search_links(request=request))
has_upstream = any(link.target.fully_qualified_name == linked_resource for link in links)
has_downstream = any(link.source.fully_qualified_name == linked_resource for link in links)
gaps = []
if not links:
gaps.append("CRITICAL_LINEAGE_GAP: No upstream or downstream lineage registered.")
elif not has_upstream:
gaps.append("ORPHAN_SOURCE_GAP: Table has downstream consumers but no documented source pipeline.")
elif not has_downstream:
gaps.append("TERMINAL_SINK_GAP: Table has upstream inputs but no registered downstream consumers.")
return {
"link_count": len(links),
"lineage_gaps": gaps,
"status": "PASS" if not gaps else "FAIL"
}
B. Taxonomy, PII & Policy Masking Detective
Create security_agent.py using Gemini 3.8 to evaluate column names, infer sensitive data types, and check against existing Dataplex Policy Tags:
python
# security_agent.py
import json
from google import genai
from google.genai import types
class SecurityTaxonomyAuditor:
def __init__(self):
self.client = genai.Client()
def audit_security_and_pii(self, table_name: str, columns: list[dict], existing_tags: list[dict]) -> dict:
"""Detects unclassified sensitive columns and missing policy tags."""
schema_context = json.dumps(columns, indent=2)
tags_context = json.dumps(existing_tags, indent=2)
prompt = f"""
You are a Cloud Data Governance and Compliance Expert inspecting Google Cloud Dataplex metadata.
Table: {table_name}
Columns:
{schema_context}
Existing Applied Tags:
{tags_context}
Tasks:
1. Identify any columns containing PII, Financial Data, Secrets, or Credentials that LACK policy/taxonomy tags.
2. Identify if column-level data masking policies appear missing for sensitive fields.
3. Output strict JSON with the following format:
{{
"unclassified_pii_columns": ["col1", "col2"],
"missing_policy_tags": ["High_Security_PII", "Mask_Email"],
"risk_level": "HIGH | MEDIUM | LOW",
"justification": "Explanation of risks"
}}
"""
response = self.client.models.generate_content(
model="gemini-2.5-pro",
contents=prompt,
config=types.GenerateContentConfig(
response_mime_type="application/json"
)
)
return json.loads(response.text)
C. Freshness, Quality & Schema Evolution Auditor
Create drift_agent.py to inspect missing business documentation and data freshness indicators:
python
# drift_agent.py
import json
from google import genai
from google.genai import types
class DriftAndQualityAuditor:
def __init__(self):
self.client = genai.Client()
def audit_documentation_and_drift(self, table_metadata: dict) -> dict:
"""Evaluates documentation completeness, empty descriptions, and business glossary alignment."""
columns = table_metadata.get("columns", [])
empty_cols = [c["name"] for c in columns if not c.get("description")]
table_desc = table_metadata.get("description", "")
prompt = f"""
Evaluate metadata completeness for database table: {table_metadata.get('display_name')}.
Table Description: '{table_desc}'
Columns with empty descriptions: {empty_cols}
Total columns: {len(columns)}
Produce recommended business descriptions for missing attributes and identify catalog gaps.
Respond in JSON:
{{
"missing_table_description": {"true" if not table_desc else "false"},
"missing_column_descriptions_count": {len(empty_cols)},
"missing_column_list": {json.dumps(empty_cols)},
"suggested_table_description": "Clear professional description if missing",
"glossary_terms_needed": ["Term1", "Term2"]
}}
"""
response = self.client.models.generate_content(
model="gemini-2.5-flash",
contents=prompt,
config=types.GenerateContentConfig(
response_mime_type="application/json"
)
)
return json.loads(response.text)
4. Multi-Agent Orchestrator & Remediation Engine
Create orchestrator.py using FastAPI. The orchestrator triggers all sub-agents, aggregates the gaps into a Catalog Health Score, and provides an automated remediation payload ready for datacatalog.create_tag:
python
# orchestrator.py
import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from catalog_client import DataplexMetadataClient
from lineage_agent import LineageAuditor
from security_agent import SecurityTaxonomyAuditor
from drift_agent import DriftAndQualityAuditor
app = FastAPI(title="Dataplex Autonomous Metadata Governance Engine")
PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-gcp-project")
metadata_client = DataplexMetadataClient(project_id=PROJECT_ID)
lineage_auditor = LineageAuditor(project_id=PROJECT_ID)
security_auditor = SecurityTaxonomyAuditor()
drift_auditor = DriftAndQualityAuditor()
class AuditRequest(BaseModel):
entry_name: str # Format: projects/{p}/locations/{l}/entryGroups/{g}/entries/{e}
@app.post("/audit-entry")
async def run_metadata_audit(request: AuditRequest):
# 1. Fetch live Dataplex metadata
meta = metadata_client.get_table_metadata(request.entry_name)
if "error" in meta:
raise HTTPException(status_code=404, detail=meta["error"])
linked_resource = meta.get("linked_resource", "")
# 2. Run Sub-Agent Audits
lineage_results = lineage_auditor.audit_lineage(linked_resource)
security_results = security_auditor.audit_security_and_pii(
table_name=meta.get("display_name", ""),
columns=meta.get("columns", []),
existing_tags=meta.get("tags", [])
)
drift_results = drift_auditor.audit_documentation_and_drift(meta)
# 3. Calculate Catalog Health Index (0-100)
penalties = 0
if lineage_results["lineage_gaps"]: penalties += 30
if security_results["unclassified_pii_columns"]: penalties += 40
if drift_results["missing_table_description"]: penalties += 15
if drift_results["missing_column_descriptions_count"] > 0: penalties += 15
health_score = max(0, 100 - penalties)
# 4. Generate Remediation Plan
return {
"entry": meta["display_name"],
"catalog_health_score": health_score,
"gaps_detected": {
"lineage": lineage_results["lineage_gaps"],
"security_and_taxonomy": security_results,
"documentation_and_glossary": drift_results
},
"remediation_action": {
"apply_dataplex_aspect_tag": {
"template": "projects/my-project/locations/us-central1/tagTemplates/governance_compliance_aspect",
"fields": {
"audit_status": "FLAGGED_FOR_REMEDIATION",
"pii_risk_level": security_results["risk_level"],
"suggested_description": drift_results["suggested_table_description"]
}
}
}
}
5. Containerization & Deployment to Cloud Run
Dockerfile
dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT=8080
CMD uvicorn orchestrator:app --host 0.0.0.0 --port ${PORT}
Deployment Commands
Enable the necessary APIs:
bash
gcloud services enable \
run.googleapis.com \
datacatalog.googleapis.com \
dataplex.googleapis.com \
datalineage.googleapis.com \
aiplatform.googleapis.com
Grant Cloud Runβs Service Account access to knowledge catalog and Vertex AI:
bash
PROJECT_ID=$(gcloud config get-value project)
PROJECT_NUM=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
SA_EMAIL="${PROJECT_NUM}-compute@developer.gserviceaccount.com"
# Grant roles for Data Catalog, Dataplex, and Vertex AI
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:${SA_EMAIL}" \
--role="roles/datacatalog.viewer"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:${SA_EMAIL}" \
--role="roles/datacatalog.tagEditor"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:${SA_EMAIL}" \
--role="roles/datalineage.viewer"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:${SA_EMAIL}" \
--role="roles/aiplatform.user"
Deploy the multi-agent service:
bash
gcloud run deploy dataplex-metadata-agent \
--source . \
--region us-central1 \
--set-env-vars "GCP_PROJECT_ID=${PROJECT_ID}" \
--no-allow-unauthenticated \
--quiet
6. Testing & Verifying Catalog Gap Remediation
Retrieve your Cloud Run service URL and trigger an audit against a target BigQuery entry in Dataplex:
bash
SERVICE_URL=$(gcloud run services describe dataplex-metadata-agent --region us-central1 --format="value(status.url)")
ENTRY_NAME="projects/${PROJECT_ID}/locations/us-central1/entryGroups/@bigquery/entries/$(echo -n "bigquery.googleapis.com/projects/${PROJECT_ID}/datasets/analytics_dw/tables/user_transactions" | base64)"
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-identity-token)" \
-H "Content-Type: application/json" \
-d "{\"entry_name\": \"${ENTRY_NAME}\"}" \
"${SERVICE_URL}/audit-entry"
Execution Output Example
JSON
{
"entry": "user_transactions",
"catalog_health_score": 30,
"gaps_detected": {
"lineage": [
"TERMINAL_SINK_GAP: Table has upstream inputs but no registered downstream consumers."
],
"security_and_taxonomy": {
"unclassified_pii_columns": ["customer_email", "billing_address", "ip_hash"],
"missing_policy_tags": ["Confidential_PII_Taxonomy/Email", "Confidential_PII_Taxonomy/Address"],
"risk_level": "HIGH",
"justification": "Direct PII stored in plain text with no column-level masking policy or IAM restriction tags attached."
},
"documentation_and_glossary": {
"missing_table_description": true,
"missing_column_descriptions_count": 8,
"suggested_table_description": "Stores user payment transaction histories, merchant identifiers, and checkout session metadata for financial reconciliation.",
"glossary_terms_needed": ["Customer_Account", "Gross_Merchandise_Value"]
}
},
"remediation_action": {
"apply_dataplex_aspect_tag": {
"template": "projects/my-project/locations/us-central1/tagTemplates/governance_compliance_aspect",
"fields": {
"audit_status": "FLAGGED_FOR_REMEDIATION",
"pii_risk_level": "HIGH"
}
}
}
}
Conclusion
Static metadata catalogs decay the moment new pipelines launch. By combining Google Cloud Knowledge Catalog with an Agentic AI governance loop, organizations can:
-
Eliminate Blindspots: Proactively detect unclassified PII, missing policy tags, and severed lineage edges before compliance audits flag them.
-
Automate Documentation: Eliminate documentation debt by generating accurate column descriptions and business glossary mappings with Gemini 3.8.
-
Close the Loop: Empower agents to directly generate Dataplex Tag templates and remediation entries, turning governance from a reactive chore into an autonomous continuous control plane.
Happy cataloging!