How to effectively serve MTP-based Gemma 4 models for inference performance

[Bharath_Rathinam]

In modern Large Language Model (LLM) serving, decoding latency is fundamentally bottlenecked by GPU memory bandwidth rather than compute. For every single token generated during the decoding phase, the serving engine must read the entire multi-billion-parameter weight matrix from High-Bandwidth Memory (HBM) to compute cores.

This guide provides actionable recommendations and tuning parameters to serve Gemma 4 models effectively on GPUs, reducing Total Cost of Ownership (TCO) without compromising output quality or reasoning performance. Our empirical evaluations were conducted on Google Cloud G4 VM instance types powered by NVIDIA RTX PRO 6000 Blackwell GPUs, though these architectural best practices apply equally across all modern GPU families supporting Multi-Instance GPU (MIG) and virtual GPU (vGPU) slicing.

---

1. Empirical performance: no MTP vs. k=1 vs. k=2

To hook into the immediate TCO and latency benefits of Gemma 4’s native Multi-Token Prediction (MTP), we evaluated decoding performance across three speculative operating regimes on Google Cloud G4 Blackwell instances.

Empirical benchmark comparison (no MTP vs. Speculative Depths)

Speculative Depth	Mean Accepted Tokens / Step	Relative Token Yield / Step	Position 0 Acceptance	Position 1 Acceptance	Relative TPOT Speedup	Relative E2E Latency Reduction
no MTP (k=0, Baseline)	1.00 tok/step	Baseline (0%)	—	—	Baseline (0%)	Baseline (0%)
k = 1	~1.45 tok/step	+45%	~47%	—	7.0% faster	7% lower
k = 2 (Optimal)	~1.71 tok/step	+71%	~47%	~24%	14.0% faster	14% lower

Key takeaways from our benchmarks

• 14% Faster Decoding without Quality Loss: Advancing from no MTP (k=0) to k=2 speculative decoding increased Mean Acceptance Length (MAL) to ~1.71 tokens per forward step (+71% yield), delivering a net 14.0% reduction in Time-per-Output-Token (TPOT) and a 14% drop in End-to-End latency.
• Why We Avoided k >= 3 on This Workload: Acceptance rates decay exponentially with position depth. Practitioners should iteratively experiment with higher speculative token counts (k=3, k=4…) on their own hardware and stop at the threshold where MAL plateaus, ensuring verification compute costs do not overrun and degrade TPOT.

---

2. Hardware slicing: Why MIG and vGPU make G4 ideal for agentic inference

Achieving high GPU utilization in multi-tenant agentic inference workloads—where multiple autonomous agent microservices share physical hardware—requires isolation against noisy-neighbor cache contention.

Google Cloud G4 machines powered by NVIDIA RTX PRO 6000 Blackwell GPUs natively support creating up to 4 Multi-Instance GPU (MIG) slices per physical GPU card, as well as fractional GPU slicing via virtual GPU (vGPU).

• Hardware-enforced isolation: Unlike generic software-level worker processes that compete for shared L2 cache and memory bandwidth, MIG and vGPU partition the GPU at the hardware and driver level.
• Predictable quality-of-service (QoS): Each of the 4 MIG/vGPU instances receives a dedicated slice of memory bandwidth and VRAM (~24 GB per slice on a 96 GB card), guaranteeing low tail latency across concurrent agentic workflows.
• 75% TCO reduction: Packing 4 isolated MTP-enabled worker instances onto a single physical G4 GPU card multiplies card QPS capacity by 4x while reducing required physical GPU count by 75%.

---

3. The Gemma 4 family and paired assistant models

Released in May 2026, the Gemma 4 family introduces a new paradigm for open-weights serving across edge devices, developer workstations, and cloud clusters. The release spans four target model sizes: gemma-4-E2B-it, gemma-4-E4B-it, gemma-4-26B-MoE-it, and gemma-4-31B-it.

A defining feature of the Gemma 4 release is that every individual target model variant comes with its own dedicated, paired MTP assistant checkpoint (e.g., google/gemma-4-E4B-it-assistant).

Figure 1: Gemma 4 MTP 4-layer drafter architecture and shared input embeddings

Credit & Source: Maarten Grootendorst, “A Visual Guide to Gemma 4” (April 2026). Reproduced for educational commentary on 4-layer MTP drafter heads and shared input embedding tables.

Why target models and assistant models are paired

To generate draft tokens efficiently without quality loss, the assistant model is designed to work in tandem with its matching target model:

• How draft tokens are generated: Instead of processing the prompt from scratch, the assistant is structured as a lightweight 4-layer transformer drafter head that takes the output representation from the target model’s top layer (h_t) to predict candidate future tokens ahead of time.
• Shared input embedding table: The MTP assistant shares the target model’s input embedding table. In large models with a 256,000-token vocabulary, embedding tables account for hundreds of megabytes of weights; sharing this table significantly reduces the VRAM footprint.
• Clustered vocabulary projections in edge models: For edge models (E2B and E4B), projecting hidden states over a massive 256k vocabulary is computationally expensive. Gemma 4 edge assistants implement a clustered embedder, grouping vocabulary tokens into hierarchical clusters to reduce logit projection math by orders of magnitude.
• How they are verified in the forward pass: Each target model size has its own internal layer dimensions and vocabulary structure. The assistant is paired to match those exact dimensions, allowing the target model to evaluate and verify all proposed draft tokens simultaneously in a single forward pass.

---

4. The evolution of multi-token prediction (MTP)

While Google researchers first introduced speculative decoding for Transformers in November 2022 (Fast Inference from Transformers via Speculative Decoding, Leviathan et al., arXiv:2211.17192), the formal training objective of Multi-Token Prediction (MTP) evolved through three key milestones:

1. Meta AI (FAIR) — April 2024: Introduced the formal MTP training methodology in Better & Faster Large Language Models via Multi-token Prediction (Gloeckle et al., arXiv:2404.19737). They demonstrated that training models to predict n future tokens simultaneously improves both sample efficiency and decoding speed.
2. DeepSeek — December 2024 / 2025: Popularized MTP at frontier scale in DeepSeek-V3 and DeepSeek-R1, proving that MTP proposer heads work effectively for self-speculative decoding in large Mixture-of-Experts (MoE) architectures.
3. Google DeepMind (Gemma 4) — May 2026: Advanced the MTP architecture for open models by releasing lightweight companion assistant checkpoints that graft directly onto target model activations with a shared KV-cache, plus clustered embedders to eliminate logit projection bottlenecks on consumer and edge hardware (E2B / E4B).

---

5. MTP vs. previous speculative decoding: Saving VRAM and KV-cache

Conventional speculative decoding frameworks rely on hosting an external, smaller “draft model” alongside the primary target LLM. While effective at reducing inter-token latency, external draft models introduce severe operational overhead in production:

• Dual-checkpoint complexity: Engineers must maintain, version, and deploy two distinct model architectures for every production service.
• Dedicated VRAM overhead: Hosting draft model weights reserves GPU memory that would otherwise support KV-cache pools or multi-tenant worker instances.
• Duplicated memory and compute: Standard external draft models maintain their own independent KV-cache and must re-encode and re-compute attention over the entire prompt from scratch.

Figure 2: Shared KV-cache pool and single-pass parallel verification flow

Credit & Source: Maarten Grootendorst, “A Visual Guide to Gemma 4” (April 2026). Reproduced for educational commentary on shared KV-cache pools and single-pass parallel verification.

How Gemma 4 MTP speculative decoding eliminates waste

Gemma 4 MTP speculative decoding deploys the tiny companion assistant model (~78M to 156M parameters) inside the same worker process and VRAM allocation as the primary target model:

• Shared KV-cache pool: The MTP assistant directly shares the target model’s KV-cache pool. Zero duplicate cache memory is allocated in VRAM.
• Zero context re-computation: The MTP assistant does not re-encode the prompt or run through base transformer layers from scratch. Instead, it takes the already-computed top-layer hidden state (h_t) directly from the target model and uses the shared KV-cache to predict candidate tokens in parallel.

Execution flow: Forward pass, drafting, and verification

1. Previous step forward pass and drafting (Step t):
   The target model processes the sequence to generate the top-layer output representation for step t. Immediately, the 4-layer MTP assistant uses this top-layer output, the shared embedding table, and the shared KV-cache to predict a sequence of candidate draft tokens (t+1, t+2 \dots) in a stacked drafter chain.
2. Next step parallel verification (Step t+1):
   On the very next step, the target model receives the proposed draft sequence. It runs a single forward pass to evaluate all drafted tokens simultaneously against its true vocabulary distribution.
3. Acceptance, rejection, and compute impact:
   • How tokens are verified: For each drafted token, if the target model’s true probability distribution agrees with the draft prediction, the token is accepted. Upon the first rejected token in the sequence, any remaining drafted tokens are discarded, and the target model generates its own correct token from that position.
   • Compute trade-off: Because LLM decoding on modern GPUs is memory-bandwidth bound, verifying several drafted tokens in parallel takes roughly the same compute time as generating a single token. However, if too many drafted tokens are rejected, the GPU wastes compute cycles evaluating unaccepted tokens, which can cause Time-per-Output-Token (TPOT) latency to increase.

---

6. Fine-tuning guidance: Preserving MTP proposer alignment

A critical failure mode when adapting Gemma 4 to specialized enterprise domains (such as SQL generation, customer support, or structured data parsing) is fine-tuning only the primary target model while ignoring the paired MTP assistant checkpoint.

The domain drift problem

If an engineering team fine-tunes the base transformer layers on custom domain vocabulary but leaves the MTP assistant un-tuned, the assistant will continue drafting general-domain tokens. Consequently:

• Verification rejection rates spike.
• Overall speculative acceptance drops below 20%.
• The inference engine pays the computational cost of MTP evaluation without gaining decode acceleration.

---

7. Production observability: MTP monitoring checklist

To ensure sustained inference efficiency in production clusters, serving infrastructure should track the following four telemetry metrics.

Note: The numerical target ranges below represent an empirical reference baseline for standard mixed prompt workloads. For structured code generation, JSON output, or repetitive text parsing, MTP acceptance rates can be significantly higher (>50%).

1. Overall Speculative Acceptance Rate (%)
   • Empirical Reference Target: 34% to 36% across all generated tokens.
   • Alert Threshold: A sustained drop below 25% signals domain drift or prompt distribution mismatch, indicating that MTP proposer heads require re-calibration or co-fine-tuning.
2. Mean Acceptance Length (MAL)
   • Empirical Reference Target: ~1.70 to 1.72 tokens/step when operating at num_speculative_tokens: 2.
3. Per-Position Acceptance Decay
   • Empirical Reference Target: Position 0 should yield ~47% acceptance, while Position 1 should yield ~24% acceptance.
   • Diagnostic Use: If Position 0 is healthy but Position 1 drops below 15%, reduce num_speculative_tokens to 1 to avoid verification compute waste.
4. Net TPOT / ITL Improvement (%)
   • Target: Ensure MTP delivers a consistent 10% to 15% net TPOT speedup compared to non-speculative execution under concurrent production traffic.

---

8. Production vLLM deployment command on Google Cloud G4 MIG

Below is the standardized vLLM launch recipe for serving Gemma 4 4B inside a dedicated Multi-Instance GPU (MIG) slice (1g.24gb) or 25% vGPU slice on a Google Cloud G4 Blackwell machine:

# Executing inside a Google Cloud G4 MIG instance (1 of 4 MIG slices per physical RTX PRO 6000 Blackwell card):
export FLASHINFER_ENABLE_SAMPLING=1
export CUDA_VISIBLE_DEVICES=0

python3 -m vllm.entrypoints.openai.api_server \
  --host 0.0.0.0 --port 9190 \
  --model /home/brathinam_google_com/models/gemma4_e4b \
  --served-model-name gemma4_e4b \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "model": "/home/brathinam_google_com/models/gemma-4-E4B-it-assistant", "num_speculative_tokens": 2}'

Key configuration notes

• --gpu-memory-utilization 0.90 (Inside MIG/vGPU): Because the MIG/vGPU hardware slice itself enforces memory isolation and bandwidth boundaries (~24 GB dedicated VRAM), the vLLM engine can safely utilize 90% of the slice’s memory pool without noisy-neighbor eviction.
• FLASHINFER_ENABLE_SAMPLING=1: Accelerates logit sampling and non-attention CUDA kernels. Because Gemma 4 uses heterogeneous attention head dimensions (head_dim=256 local, global_head_dim=512 global), vLLM forces TRITON_ATTN for attention math while FlashInfer optimizes the sampling pipeline.
• --speculative-config: Points directly to the companion MTP assistant checkpoint (gemma-4-E4B-it-assistant) with num_speculative_tokens: 2, enabling shared KV-cache speculative verification for a 14% TPOT decode latency reduction.