Low implicit cache hit rate on Vertex AI MaaS GLM 5 despite byte-stable system prompt and tools

Google Cloud Build with AI Generative AI & Foundational Models
1

We are using GLM 5 MaaS on Vertex AI for a production agentic workload with sequential tool-calling turns.

Our request prefix is stable across turns:

  • System prompt is byte-stable.

  • Tool definitions are byte-stable.

  • Tool ordering is deterministic.

  • No dynamic timestamps, request IDs, or per-turn metadata are inserted before the stable prefix.

  • Requests are sequential, not parallel.

Despite that, our observed implicit cache hit rate in production is only around 18% to 22%.

Example usage pattern:

agent_1:
input_tokens: ~7k
cached_tokens: mostly 0
occasional hit: ~5.5k cached

agent_2:
input_tokens: ~28k-32k
cached_tokens: mostly 0
occasional hit: ~31k cached
then misses again on later sequential turns

This suggests implicit caching is working sometimes, but cache reuse is not deterministic enough for an agentic workload with large static prefixes.

Questions:

  1. For GLM 5 MaaS, is implicit cache scoped per global endpoint, per region, per internal cluster, or per serving replica?

  2. If we use the OpenAI-compatible endpoint with a regional location path, for example:
    https://aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/us-east4/endpoints/openapi/chat/completions

does that pin requests to one regional cache domain, or can requests still route across multiple independent cache domains inside the region?

  1. Is there any supported cache affinity, session affinity, routing key, or conversation-level stickiness for GLM 5 MaaS?

  2. Does Provisioned Throughput improve cache locality, or does it only affect quota/priority?

  3. Is explicit context caching supported or planned for GLM 5 MaaS? We tried to use explicit caching and it appears unsupported/not allowed.

Are cached token metrics for GLM 5 MaaS considered reliable through the OpenAI-compatible API?

We sometimes see:
prompt_tokens_details.cached_tokens

but many responses return no cached-token details even when the prefix is unchanged.

The production problem is that implicit cache currently behaves like best-effort optimization rather than something we can rely on for latency/cost planning. For large agent tool surfaces, this makes GLM 5 MaaS hard to size accurately even when the prompt and tools are fully stable.

Any guidance from Google on cache domain, regional pinning, or planned explicit cache support for MaaS models would be helpful.

1 Like