Background: The Autonomous Agent Memory Paradox
As enterprises transition from single-turn chat assistants to fully autonomous, multi-turn reasoning loops using the Google Cloud Agent Development Kit (ADK) and Model Context Protocol (MCP), infrastructure and AI architects face a critical scalability barrier: Context Window Bloat and Token Inflation.
In complex agent workflows—such as recursive SQL tuning against BigQuery, multi-step ERP data reconciliation, or iterative code debugging—an autonomous agent routinely executes 10 to 30 tool calls per objective. Each iteration appends:
- Large JSON-RPC tool parameters.
- Raw tabular outputs and verbose error stack traces.
- Model chain-of-thought scratchpads and intermediate plans.
Within just 5 turns, session payload sizes can balloon from 2K tokens to well over 80K tokens.
The Three Operational Bottlenecks of Unbounded Context
- Exponential Latency & TTFT Degradation: Even with high-throughput foundation models like Gemini 3.1 Pro, processing massive conversational prefixes on every inference turn causes significant Time-To-First-Token (TTFT) degradation and sluggish UI responsiveness.
- Attention Dilution & Goal Drift: Saturating the context window with raw diagnostic dumps and intermediate database schemas weakens the model’s attention mechanism. In later turns, the agent frequently loses track of the initial user objective or enters circular, repetitive tool invocation loops.
- Severe FinOps Inefficiency: Token consumption scales quadratically over recursive cycles. Resending 50K+ tokens of raw intermediate history on turn 15 for a 200-token answer creates massive, unnecessary operational costs.
Three Architectural Patterns for Agent Memory Compaction
To solve this, we are evaluating three distinct memory compaction patterns inside Google ADK:
Pattern 1: Rolling Sliding Window with Semantic Pinning
- Mechanism: Retain the original system instructions, initial user objective, and the last N turns (typically 3–4 turns) verbatim.
- Pros: Simplest to implement; guarantees near-zero latency overhead for memory management.
- Cons: Loses intermediate context if the agent needs to reference an artifact discovered 6 turns prior.
Pattern 2: Recursive Hierarchical Summarization (Dual-Agent Model)
- Mechanism: When session history exceeds a pre-set threshold (e.g., 20,000 tokens), trigger a lightweight auxiliary model (such as Gemini 3.5 Flash or Gemma 2) to condense turns 1 through N-2 into a structured, delta-state briefing.
- Pros: Preserves critical state, business entities, and tool outcomes without retaining verbose raw JSON payloads.
- Cons: Introduces a periodic 400ms–800ms summarization hop between reasoning turns.
Pattern 3: Semantic Vector Eviction (Offloading to Vertex AI Vector Search)
- Mechanism: Prune tool output payloads from the active session context entirely, writing full execution payloads into an ephemeral session index in Vertex AI Vector Search or Firestore. Inject an MCP retrieval tool so the agent can query past tool outputs on-demand.
- Pros: Keeps prompt size minimal and deterministic while maintaining infinite historical recall.
- Cons: Requires the agent to learn when to query its own past execution memory.
Comparison of Memory Compaction Approaches
| Compaction Strategy | Token Reduction | Latency Overhead | Context Fidelity | Implementation Complexity |
|---|---|---|---|---|
| Naive (Full History) | 0% | High (Scales with turns) | 100% | None |
| Sliding Window + Pinning | 60% – 80% | None (<10ms) | Low – Medium | Low |
| Hierarchical Summarization | 75% – 85% | Medium (~600ms) | High (Structured) | Medium |
| Vector Eviction + Retrieval | 85% – 95% | Low – Medium (~150ms) | High (Dynamic) | High |
Prototype ADK Session Compaction Hook
Here is an illustrative implementation exploring an automated pruning hook integrated into a custom ADK SessionService:
from google.adk.sessions import BaseSessionService, Session
from google.genai import types
class CompactedSessionService(BaseSessionService):
def __init__(self, base_service: BaseSessionService, max_turn_tokens: int = 16000):
self._inner = base_service
self._max_turn_tokens = max_turn_tokens
async def append_and_compact(self, session: Session, new_turn: types.Content) -> Session:
# 1. Append newest turn to session state
session = await self._inner.append_event(session, new_turn)
# 2. Estimate token count across active history
total_tokens = sum(len(part.text or "") // 4 for event in session.history for part in event.parts)
# 3. Trigger compaction if threshold exceeded
if total_tokens > self._max_turn_tokens:
session = await self._compact_intermediate_traces(session)
return session
async def _compact_intermediate_traces(self, session: Session) -> Session:
"""
Preserves System Instructions and User Prompt,
stripping verbose MCP JSON-RPC returns older than 3 turns.
"""
retained_history = []
for idx, event in enumerate(session.history):
# Always preserve first prompt and recent 3 turns
if idx == 0 or idx >= len(session.history) - 3:
retained_history.append(event)
else:
# Mask verbose function responses to metadata-only summaries
masked_parts = [
types.Part(text=f"[Output truncated: {len(p.text or '')} chars omitted for brevity]")
if hasattr(p, "function_response") else p
for p in event.parts
]
retained_history.append(types.Content(role=event.role, parts=masked_parts))
session.history = retained_history
return session
Questions for the Community & Google Cloud Team
- Native ADK Pruning Hooks: Are there roadmap plans for built-in memory pruning or summarization middlewares within the core
RunnerorAgentlifecycle in ADK? - Context Compression vs. Implicit Caching: With Gemini’s native context caching, is it more cost-effective to let large contexts remain cached rather than spending compute cycles compacting history?
- MCP Tool Response Scrubbing: How are production teams handling recursive MCP tool loops without polluting reasoning context with massive tabular query returns?
Looking forward to hearing architectural perspectives and production experiences from other teams building on Google Cloud!