Authors
- Rohit Sroch (AI Engineer, Google Cloud)
- Theo Haddad (AI Engineer, Google Cloud)
- Sriram Srinivasan (AI Consultant, Google Cloud)
- Ajit Sonawane (AI Engineer, Google Cloud)
In most enterprise AI systems today, an agent’s behavior—its root system instructions and modular domain skills (SKILL.md)—is shipped as static source code inside the application repository. While treating prompts as application code works for early prototypes, in production the real world drifts as lending guidelines, credit rules, and edge cases evolve, creating three major operational bottlenecks:
-
The monolithic prompt optimization ceiling: Standard prompt optimizers only rewrite a single system instruction, so on a skill-based agent they cram domain math and policy rules into the root routing prompt instead of the
SKILL.mdfiles where those rules belong. -
Cross-skill regressions (“whack-a-mole” prompt surgery): Editing one prompt or skill to fix an edge case (like a debt-to-income rule) often pulls the model’s attention away from another capability (like collateral valuation), and binary
PASS/FAILchecks throw away a whole optimization run over one local dip. -
Full container redeployments for behavioral fixes: Because prompts and
SKILL.mdfiles sit in the container image, a two-line rule change requires a pull request, CI/CD build, and redeploy—with no fast rollback if the new instruction misbehaves.
To solve these challenges, this article shows how to build a closed-loop, self-optimizing agent lifecycle (AgentOps) on Google Cloud using the Agent Development Kit (ADK), the GEPA optimizer, the Gen AI Evaluation Service, Agent Platform Prompt Management (built on Agent Platform Datasets), and Agent Platform Experiments.
This framework-agnostic pattern applies to any agent architecture—whether built with ADK, LangGraph, or custom code—whose behavior lives in a system instruction plus SKILL.md files. We walk through it with a mortgage underwriting agent evaluated on multi-document underwriting submissions.
Treat agent instructions and modular
SKILL.mdfiles as governed runtime configuration rather than static code. Co-evolve prompts and skills together using structured evaluator feedback, gate candidates with an automated surgical-patch analyzer, and hot-load promoted versions at runtime from a versioned registry—with zero container redeployments and two-second rollbacks.
The diagram below illustrates the five-stage self-optimizing lifecycle:
Fig 1: The five-stage self-optimizing lifecycle, from production traces and prompt + skill co-optimization to gap analysis, managed prompt and skill registries, and the zero-redeploy runtime
Separating evolving behavior from static agent code
In mortgage underwriting, before a lender clears a loan to close on a single-family home, condominium, or multi-unit residential property, an underwriter has to cross-check the borrower’s loan application against credit reports, tax returns, bank statements, and property appraisals—flagging hidden liabilities or collateral issues, requiring additional reserves or stipulations, and escalating high-risk exceptions to a senior credit officer.
Our showcase Mortgage Underwriting Agent automates drafting that credit memo. Given a loan application_id, the root mortgage_underwriting_agent delegates to three single-turn specialist sub-agents—each isolated to its own document-retrieval tool and domain SKILL.md—and assembles their findings into a decision-ready, three-section Markdown report.
Fig 2: Mortgage Underwriting Agent architecture. A root mortgage_underwriting_agent delegates to credit profile, income and capacity, and property collateral specialists, each with isolated document lookup tools and modular
SKILL.md files
To keep responsibilities cleanly separated, each specialist uses a thin, fixed role instruction (“load your skill, retrieve your section’s source documents, and return the drafted section”). All evolving behavior lives outside the container image across two dedicated Agent Platform services: the root orchestrator prompt (governing sub-agent delegation and report assembly) is versioned in Agent Platform Prompt Management, while the three specialist SKILL.md packages (defining each section’s domain thresholds, document-precedence rules, and Markdown structure) are versioned in Skill Registry within Agent Registry.
Using ADK’s InstructionProvider callable and native GCPSkillRegistry client, the agent resolves active versions dynamically at runtime instead of baking static files into the container at startup:
import vertexai
from google.adk.agents import Agent
from google.adk.agents.readonly_context import ReadonlyContext
from google.adk.integrations.skill_registry.gcp_skill_registry import GCPSkillRegistry
from google.adk.tools.skill_toolset import SkillToolset
vx_client = vertexai.Client(project=PROJECT_ID, location=LOCATION)
skill_registry = GCPSkillRegistry(project_id=PROJECT_ID, location=LOCATION)
def get_root_instruction(ctx: ReadonlyContext) -> str:
"""ADK InstructionProvider: resolves the active root prompt via Prompt Management."""
prompt = vx_client.prompts.get(prompt_id=ROOT_PROMPT_ID)
return prompt.prompt_data.system_instruction.parts[0].text
def build_root_agent() -> Agent:
credit_profile_agent = Agent(
name="credit_profile_agent",
model="gemini-3.8-flash",
mode="single_turn",
instruction=CREDIT_SPECIALIST_ROLE_INSTRUCTION, # Fixed thin role instruction
tools=[
get_credit_docs,
SkillToolset(registry=skill_registry), # Fetches active `SKILL.md` revision from Agent Registry
],
)
# income_capacity_agent and property_collateral_agent follow the exact same pattern...
return Agent(
name="mortgage_underwriting_agent",
model="gemini-3.8-flash",
instruction=get_root_instruction, # Resolves active prompt_id via Prompt Management
sub_agents=[credit_profile_agent, income_capacity_agent, property_collateral_agent],
)
Preparing evaluation splits for policy drift
To test how the lifecycle handles real policy drift, our baseline v1 uses 2025 underwriting rules and outputs free-form paragraphs, while incoming 2026 submissions are graded against tightened 2026 guidelines (like higher minimum credit-score thresholds (680 vs. 640), tighter debt-to-income caps (43% vs. 50%), narrower appraisal-variance tolerances (5% vs. 10%), and structured bullet points).
We export production traces and expert-reviewed 2026 reference reports from BigQuery into 400 submissions (300 train, 50 validation, 50 held-out test). With three sections per report, that gives us 1,200 section-level JSONL records:
{
"section": "1. Credit History & Liabilities",
"application_id": "LN-2026-001",
"location": "Austin, TX",
"source_context": "[URLA Form 1003 Loan Application] Loan #LN-2026-001 (Austin, TX). Occupancy: Primary Residence, Single-Family Detached. Requested loan amount: $480,000 ...\n[Tri-Merge Credit Report #CR-202601] Representative middle FICO score: 668. Revolving utilization at 34%; undisclosed $420/mo auto lease opened 45 days ago...",
"reference_section": "## 1. Credit History & Liabilities\n\n* **Credit Profile Analysis**:\n - Summary: Loan #LN-2026-001 (Austin, TX, $480,000 primary residence); middle FICO 668 with 34% revolving utilization...\n - Score & Liability Exposure: Middle FICO (668) falls below the 2026 680 minimum; include $420/mo lease in DTI...\n - Mandatory Referral: Refer to Senior Underwriter — ..."
}
At runtime, each specialist’s lookup tool returns only the submission metadata and source_context—stripping reference_section so the agent can never peek at ground truth—while the evaluator uses reference_section solely to grade each section.
Why prompt-only optimization hits a ceiling—and how co-evolving skills breaks it
When agent quality drops after a policy change, the first instinct is usually to run Vertex AI Prompt Optimizer on the root system prompt. That works well for single-prompt agents, where one instruction controls everything. In a multi-agent system with isolated tools and external SKILL.md files, tuning only the root prompt quickly hits two walls:
-
The root agent never sees the raw documents. Only the specialists call the lookup tools and read the underlying credit reports, income statements, and appraisals. Even if you perfect the root prompt’s formatting rules, the specialists still apply their frozen 2025
SKILL.mdlogic—missing new 2026 risks before the root agent ever sees their drafts. -
Cramming domain rules into the root prompt breaks modularity. Moving specialist math and referral rules up into the orchestrator bloats its routing prompt, and still can’t recover facts a sub-agent already left out.
Fig 3: Monolithic prompt-only optimization hits a ceiling when sub-agent SKILL.md files stay frozen, whereas component-attributed GEPA co-evolution routes section-level judge feedback directly to the specific prompt or SKILL.md responsible—such as credit-profile, income-capacity, or property-collateral
To fix the rules where they actually live, we use GEPA to evolve all four components together: the root orchestrator prompt (instruction:mortgage_underwriting_agent) and the three specialist SKILL.md bodies (keeping their YAML frontmatter intact).
After the agent drafts a report, the Agent Platform Gen AI Evaluation Service grades each section against the 2026 reference report on a 1–5 scale across four criteria—completeness, coherence, conciseness, and correctness—returning both a score and a plain-text explanation. Our adapter combines those explanations with ADK execution traces so each file only gets feedback for mistakes it actually caused:
-
Root orchestrator prompt: Gets feedback on missed sub-agent calls, coherence drops, or cases where it rewrote a specialist’s clean bullet points back into paragraphs.
-
Specialist
SKILL.mdfiles: Get their own section’s completeness, correctness, and conciseness scores and explanations (for example, applying the outdated 640 credit-score minimum instead of 680, or omitting a newly opened auto lease from the debt-to-income calculation).
import gepa
result = gepa.optimize(
seed_candidate={
"instruction:mortgage_underwriting_agent": root_orchestrator_prompt,
"skill:credit-profile": credit_skill_markdown_body,
"skill:income-capacity": income_skill_markdown_body,
"skill:property-collateral": collateral_skill_markdown_body,
},
trainset=train_applications,
valset=val_applications,
adapter=eval_adapter, # Runs ADK agent + grades sections via Gen AI Evaluation Service
reflection_lm=gemini_lm, # Gemini model GEPA uses to rewrite one prompt/SKILL.md per round
module_selector="round_robin",
max_metric_calls=400, # Bounded optimization budget across the validation split
)
In each optimization round, GEPA runs the agent on a small training batch, grades the output with the Gen AI Evaluation Service, and uses Gemini (reflection_lm) to rewrite one entry in seed_candidate at a time—either root_orchestrator_prompt or one of the three SKILL.md files (credit-profile, income-capacity, or property-collateral). Updating one component per round keeps 2026 domain rules inside each skill and produces GEPA’s top candidate bundles (prompt + 3 SKILL.md).
Two-layer quote-verified gap analysis: promote, patch, or re-optimize
Because GEPA optimizes over small batches using an overall average score, even a strong candidate can still have a small blind spot—like missing a rare 2026 edge case, or trimming a required referral detail while trying to be concise. Looking only at the average score hides those local dips, while a strict pass/fail gate throws away a great optimization run over a three-line fix in one SKILL.md.
Instead, we regenerate reports for each candidate bundle on the held-out split, score them with the Gen AI Evaluation Service, and pass the results into a Two-Layer Gap Analysis Service:
Fig 4: The Two-Layer Gap Analysis Service combines deterministic 4-C metric and rationale extraction (Layer 1) with quote-verified Gemini root-cause diagnosis comparing generated vs. reference reports (Layer 2) to route candidates to promote, patch, or re-optimize
-
Layer 1 — Deterministic ranking and score checks: Computes average
1–5scores per criterion and per section across candidates, selects the winning candidate bundle (prompt + 3 SKILL.md) by composite score, compares it againstv1, and pulls out any remaining low-scoring sections along with their judge explanations, generated text, and reference sections—without making any LLM calls. -
Layer 2 — Quote-verified root-cause diagnosis: On that winning candidate bundle (
prompt + 3 SKILL.md), uses Gemini to compare the generated and reference sections against the source documents, judge explanations, and prompt/skill diffs—separating missing or regressed rules from missing input data (so teams don’t rewrite skills when a fact wasn’t insource_context). Every finding must cite exact quotes from the prompt orSKILL.mdfiles, verified in code to prevent hallucinated fixes before choosing one of three actions:-
promote: Clears all four score thresholds with no regressions; ready for promotion to the registries and underwriter sign-off. -
patch: Fixes the main 2026 drift but leaves a small gap in one or two files; applies a targeted edit to just that component and re-scores on the held-out split. -
re_optimize: Misses targets across multiple skills; sends the structured diagnosis back into GEPA for another round.
-
Zero-redeploy promotion from managed prompt and skill registries
In a typical ADK deployment on Cloud Run or Agent Platform Agent Engine, the Agent and Runner initialize once when the container starts. If prompts and SKILL.md files are baked into the container image, even a two-line rule update requires a pull request, image build, and redeploy.
To ship an approved v2 without rebuilding or restarting containers, we store the prompt and skills in managed Agent Platform registries and load them at runtime:
Fig 5: Managed versioning across Agent Platform Prompt Management (root prompt) and Agent Registry’s Skill Registry (
SKILL.md revisions), version governance, ADK runtime loaders, and zero-redeploy execution with Cloud Trace lineage
-
Store prompts and skills as versioned assets: The root prompt lives in Agent Platform Prompt Management, while each specialist
SKILL.mdis stored in Agent Registry’s Skill Registry—paired with its held-out 4-C scorecard in Agent Platform Experiments. -
Promote or roll back in seconds: Shipping
v2simply publishes a new prompt version and updates each skill’s active revision. If live monitoring catches an issue, pointing back to v1 restores previous behavior immediately—no container rebuilds required. -
Hot-reload safely at runtime: Running containers fetch active versions through
get_root_instruction(an ADKInstructionProvidercallable that loads the prompt byprompt_idfrom Prompt Management) andGCPSkillRegistry(which loads activeSKILL.mdrevisions from Skill Registry) behind a 60-second in-memory cache with a last-known-good fallback, stamping the exact prompt and skill version IDs onto every Cloud Trace span.
Putting it all together on Agent Platform
In production, these pieces connect into one closed-loop pipeline on Google Cloud Agent Platform. The workflow builds the Train / Validation / Test splits from BigQuery traces, generates candidate prompt and SKILL.md bundles with GEPA, re-runs the ADK Underwriting Agent per candidate, selects and gates the winning bundle through Two-Layer Gap Analysis (re_optimize, patch, or promote), and publishes the winner to Prompt Management and Agent Registry for live hot-reloading.
Once live, continuous drift monitoring streams production traces from the Live ADK Agent back into BigQuery—feeding Prepare Evaluation Splits whenever real-world submissions or underwriting guidelines shift so the agent keeps adapting over time:
Fig 6: End-to-end self-optimizing pipeline on Agent Platform connecting evaluation splits, GEPA prompt and skill co-optimization, ADK report regeneration, Two-Layer Gap Analysis, zero-redeploy promotion to Prompt Management and Agent Registry, and continuous drift monitoring closing the loop
Measuring each layer on held-out underwriting submissions
We evaluated all four configurations on the 50 held-out 2026 mortgage loan submissions (150 total sections) using gemini-3.8-flash for the multi-agent report generator and the Agent Platform Gen AI Evaluation Service (1–5 scale across the four criteria):
Fig 7: Held-out underwriting benchmark across 50 test submissions (150 evaluated sections) comparing hand-written v1, prompt-only GEPA, full prompt +
SKILL.md GEPA co-evolution, and GEPA + Two-Layer Gap Analysis patch (v2)
| Configuration | Completeness | Coherence | Conciseness | Correctness | Composite (4-C Average) | Gate outcome / rollout |
|---|---|---|---|---|---|---|
1. Hand-written v1 (2025 rules) |
2.67 |
1.22 |
2.91 |
2.93 |
2.43 |
Baseline (days: PR + redeploy) |
| 2. Prompt-only GEPA (skills frozen) | 3.09 |
4.93 |
4.98 |
3.13 |
4.03 |
Blocked (re_optimize) |
3. Full GEPA (prompt + 3 SKILL.mds) |
4.59 |
4.91 |
4.79 |
4.58 |
4.72 |
Held for targeted fix (patch) |
4. Full GEPA + Gap Analysis v2 |
4.96 |
5.00 |
4.91 |
4.93 |
4.95 |
Promoted (promote, 0 redeploys) |
Three takeaways stand out in the held-out traces:
-
Prompt-only optimization fixes formatting, then hits a domain ceiling. Tuning only the root prompt lifted coherence (1.22 → 4.93) and conciseness (2.91 → 4.98) by teaching the orchestrator to assemble structured bulleted sections without extra introductory text (composite 2.43 → 4.03). However, completeness (3.09) and correctness (3.13) stayed low because the frozen
SKILL.mdfiles still applied outdated 2025 domain rules—triggering re_optimize. -
Co-evolving prompts and skills unlocks domain accuracy. Optimizing the root prompt alongside all three
SKILL.mdfiles raised the composite score to 4.72 (completeness 4.59, correctness 4.58). Routing judge feedback to the specific file at fault allowed each specialist skill to adopt the updated 2026 rules while keeping the root prompt focused on clean orchestration. -
Targeted gap analysis closes remaining gaps without starting over. Because batch-based optimization can still leave an isolated gap in one component (for example, when a single skill update is skipped on a small batch), Full GEPA finished with two specialist sections near 4.9+ and one lagging at 4.31. Instead of discarding a 4.72 candidate, Two-Layer Gap Analysis pinpointed that single
SKILL.md, applied a targeted patch, and lifted v2 to 4.95 composite—clearing promote with zero regressions and zero container redeploys.
Get started
Treating prompts and SKILL.md files as versioned runtime assets turns policy drift from a manual rewrite cycle into a routine AgentOps loop. To build this workflow for your own agents:
-
Build and co-optimize modular agents: Wire single-turn specialists with the Agent Development Kit (ADK) and evolve prompts alongside
SKILL.mdfiles using GEPA. -
Externalize prompts and skills: Version orchestrator prompts in Agent Platform Prompt Management and specialist skills in Agent Registry’s Skill Registry for zero-redeploy updates.
-
Automate evaluation, gating, and drift monitoring: Score held-out traces with the Gen AI Evaluation Service, and stream live agent traces through Cloud Trace and BigQuery to detect drift and feed the next evaluation cycle.







