Deploying Moonshot AI’s Kimi K3—a 2.8-trillion-parameter Mixture-of-Experts (MoE) model built on novel Kimi Delta Attention (KDA) and Stable LatentMoE routing—defines the next level of scale for open-weights AI engineering.
However, serving a model of this size is an operational challenge. While MoE models are computationally efficient—activating only 16 of 896 experts per token—their physical footprint requires massive memory bandwidth and capacity. More importantly, serving these models under high concurrency without stalling accelerators requires extreme network throughput and highly optimized decoding algorithms.
Today, we are laying out how Google Cloud is the best infrastructure for deploying and serving Kimi K3, highlighting flexible deployment paths from Model Garden to Kubernetes-native distributed serving.
Flexible Deployment Paths on Google Cloud
We support three primary pathways to serve Kimi K3, catering to different operational needs:
-
Model Garden on Gemini Enterprise Agent Platform (Fastest Path): The easiest path for enterprises. Model Garden provides one-click deployment templates, abstracting away the underlying node configuration, driver setup, and container management while providing a secure, production-grade endpoint. Explore Model Garden
-
AI Hypercomputer recipes (Validated Infrastructure): For teams running custom orchestration. Our AI Hypercomputer recipes in github validate the end-to-end stack (accelerators, storage mount configurations, and model servers) on Google Cloud infrastructure. See the recipes on GitHub
-
GKE with llm-d recipes (Distributed Serving): For platforms requiring immense scale and deep customization. The llm-d CNCF sandbox project integrates Kubernetes-native orchestration with advanced inference features like intelligent KV-aware routing, disaggregated prefill-decode serving, and multi-tier KV cache offloading. To learn more about llm-d, see the GitHub repository. We expect the team to update the Wide EP / LeaderWorkerSet well-lit path to include Kimi K3 shortly.
Architecture: Multi-Node Serving with SGLang and DSPARK
To achieve sub-second latency for Kimi K3, we utilize SGLang as the model server. SGLang provides native support for Kimi Delta Attention (KDA) and Stable LatentMoE, enabling highly optimized kernel execution.
Because Kimi K3’s 2.8T parameters exceeds the High Bandwidth Memory (HBM) capacity of most single machines, we deploy the model in a multi-node configuration spanning 2 nodes (A4 VM) and 4 nodes (A4X VM) with Tensor Parallelism (TP) of 16.
The recommended deployment is to run the model using the NVLink domains of A4X (GB200 NVL72) and A4X Max (GB300 NVL72) to accelerate chip to chip communication for multi-node inference.
Alternatively, we can deploy this using RDMA available on A4 VMs (B200). This will accelerate all-reduce communication across GPUs to overcome (non-RDMA) internetworking between VMs, as this absence leads to lower token throughput.
Deploying Kimi K3 on GKE
Moonshot’s Kimi K3 is subject to specific license terms and restrictions. Before deploying this model, please review these items and ensure your deployment is compliant: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE.
For GB200 deployment, refer to this guide to deploy the model across a NVL72 domain across 16 x GB200 on 4 x A4X VMs.
For B200 deployment, see below instructions.
Pre-upload Model Weights and Draft Model to GCS bucket
Create a new GCS bucket and download/upload our weights and draft model
# 1. Install Hugging Face CLI and high-speed transfer library
pip install -U huggingface_hub[cli] hf_transfer
# 2. Download the Kimi-K3 model checkpoint locally
export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_TOKEN=<yourKey>
huggingface-cli download moonshotai/Kimi-K3 --local-dir ./models--moonshotai--Kimi-K3
huggingface-cli download RadixArk/Kimi-K3-DSpark --local-dir ./models--RadixArk--Kimi-K3-DSpark
# 3. Upload the checkpoint directory to your Cloud Storage bucket
gcloud storage cp -r ./models--moonshotai--Kimi-K3 gs://${GCS_BUCKET}/huggingface_model_cache/
gcloud storage cp -r ./models--RadixArk--Kimi-K3-DSpark gs://${GCS_BUCKET}/huggingface_model_cache/
Deploying a GKE cluster
To provision a best practices AI-optimized GKE cluster with RDMA, we recommend using the guide to deploy using Cluster Toolkit or a custom deployment with gcloud CLI.
Deploying Kimi K3 with SGLang on GKE with B200
Below is the verified Kubernetes manifest to deploy Kimi K3 using SGLang with DSPARK speculative decoding on GKE. This configuration targets GKE nodes equipped with NVIDIA B200 GPUs, leveraging GPUDirect RDMA over a RoCEv2 network for high-throughput multi-node interconnect.
apiVersion: v1
kind: Service
metadata:
name: sglang-master-pod-k3
namespace: default
spec:
type: ClusterIP
clusterIP: None
selector:
app: distributed-sglang-k3
apps.kubernetes.io/pod-index: "0"
ports:
- name: dist-port
port: 20000
targetPort: 20000
---
apiVersion: v1
kind: Service
metadata:
name: sglang-serving-k3
namespace: default
labels:
app: distributed-sglang-k3
spec:
type: ClusterIP
selector:
app: distributed-sglang-k3
apps.kubernetes.io/pod-index: "0"
ports:
- name: http
port: 30000
targetPort: 30000
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: distributed-sglang-k3
namespace: default
spec:
serviceName: "sglang-master-pod-k3"
replicas: 2
podManagementPolicy: Parallel
selector:
matchLabels:
app: distributed-sglang-k3
template:
metadata:
labels:
app: distributed-sglang-k3
llm-d.ai/guide: optimized-baseline
annotations:
gke-gcsfuse/volumes: "true"
networking.gke.io/default-interface: 'eth0'
networking.gke.io/interfaces: |
[
{"interfaceName":"eth0","network":"default"},
{"interfaceName":"eth2","network":"rdma-0"},
{"interfaceName":"eth3","network":"rdma-1"},
{"interfaceName":"eth4","network":"rdma-2"},
{"interfaceName":"eth5","network":"rdma-3"},
{"interfaceName":"eth6","network":"rdma-4"},
{"interfaceName":"eth7","network":"rdma-5"},
{"interfaceName":"eth8","network":"rdma-6"},
{"interfaceName":"eth9","network":"rdma-7"}
]
spec:
serviceAccountName: sglang-sa
containers:
- name: sglang-container
image: docker.io/lmsysorg/sglang:kimi-k3
imagePullPolicy: Always
command:
- /bin/bash
- -c
args:
- |
set -ex
ldconfig /usr/local/nvidia/lib64/ || true
export LD_LIBRARY_PATH="/usr/local/gib/lib64:/usr/local/nvidia/lib64:$LD_LIBRARY_PATH"
if [ -f "/usr/local/gib/scripts/set_nccl_env.sh" ]; then
source /usr/local/gib/scripts/set_nccl_env.sh
fi
export NCCL_IB_GID_INDEX=3
export NCCL_IB_ROCE_VERSION_NUM=2
export GLOO_SOCKET_IFNAME=eth0
export NCCL_SOCKET_IFNAME=eth0
export SGLANG_HOST_IP=$(hostname -I | tr ' ' '\n' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -n 1)
echo "=========================================================="
echo "Launching SGLang multi-node for moonshotai/Kimi-K3..."
echo "Node Rank: $POD_INDEX | dist-init-addr: sglang-master-pod-k3:20000"
echo "=========================================================="
sglang serve \
--trust-remote-code \
--model-path /bucket/Kimi-K3 \
--served-model-name moonshotai/Kimi-K3 \
--load-format runai_streamer \
--enable-metrics \
--tp-size 16 \
--nnodes 2 \
--node-rank $POD_INDEX \
--dist-init-addr sglang-master-pod-k3:20000 \
--mem-fraction-static 0.85 \
--disable-flashinfer-autotune \
--watchdog-timeout 3600 \
--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--model-loader-extra-config '{"memory_limit": 34359738368}' \
--mamba-full-memory-ratio 0.43 \
--host 0.0.0.0 \
--port 30000 \
--speculative-algorithm DSPARK \
--speculative-draft-model-path /root/.cache/huggingface/Kimi-K3-DSpark \
--speculative-dspark-block-size 7 \
--enable-linear-replayssm-spec \
--max-running-requests 512 \
--cuda-graph-max-bs 512
env:
- name: POD_INDEX
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.labels['apps.kubernetes.io/pod-index']
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-secret
key: HF_TOKEN
optional: true
- name: LD_LIBRARY_PATH
value: /usr/local/nvidia/lib64
- name: NCCL_DEBUG
value: INFO
- name: SGLANG_REQUEST_STATE_WAIT_TIMEOUT
value: "18000"
- name: OMP_NUM_THREADS
value: "24"
- name: SGLANG_SET_CPU_AFFINITY
value: "1"
- name: SAFETENSORS_FAST_GPU
value: "1"
- name: PYTORCH_CUDA_ALLOC_CONF
value: "expandable_segments:True"
- name: RUNAI_STREAMER_MEMORY_LIMIT
value: "34359738368"
- name: RUNAI_STREAMER_CONCURRENCY
value: "16"
- name: SGLANG_USE_RUNAI_MODEL_STREAMER
value: "true"
resources:
limits:
nvidia.com/gpu: "8"
cpu: "206"
memory: "1500Gi"
requests:
nvidia.com/gpu: "8"
cpu: "206"
memory: "1500Gi"
volumeMounts:
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cache/huggingface
name: hf-cache
- name: library-dir-host
mountPath: /usr/local/nvidia
- name: gib-nccl-plugin-volume
mountPath: /usr/local/gib
- name: gcs-fuse-csi-eph
mountPath: /bucket
nodeSelector:
cloud.google.com/gke-accelerator: nvidia-b200
tolerations:
- operator: "Exists"
key: "nvidia.com/gpu"
- operator: "Exists"
key: "cloud.google.com/impending-node-termination"
- operator: "Exists"
key: "user-workload"
volumes:
- name: library-dir-host
hostPath:
path: /home/kubernetes/bin/nvidia
- name: gib-nccl-plugin-volume
hostPath:
path: /home/kubernetes/bin/gib
- emptyDir:
medium: Memory
sizeLimit: 100Gi
name: dshm
- name: hf-cache
hostPath:
path: /mnt/stateful_partition/kube-ephemeral-ssd/huggingface_cache
type: DirectoryOrCreate
- name: gcs-fuse-csi-eph
csi:
driver: gcsfuse.csi.storage.gke.io
volumeAttributes:
bucketName: ikwak-models-gpu-launchpad-playground
gcsfuseMetadataPrefetchOnMount: "true"
mountOptions: "implicit-dirs,file-mode=777,dir-mode=777,file-cache:enable-parallel-downloads:true,file-cache:cache-file-for-range-read:true,file-cache:max-size-mb:-1,write:enable-streaming-writes:true,read_ahead_kb=1024,file-system:kernel-list-cache-ttl-secs:-1"
Testing our model
To send an example request to the model, we can send a curl request to a local endpoint from the container.
kubectl exec -i distributed-sglang-k3-0 -c sglang-container -- \
curl -s -X POST http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "moonshotai/Kimi-K3",
"messages": [
{"role": "user", "content": "What is Google Cloud AI Hypercomputer in 2 sentences."}
],
"max_tokens": 256,
"temperature": 0.7
}' | jq .
Moving to Production Readiness
As you transition your Kimi K3 deployment from this guide into a production environment, optimizing your infrastructure for cost, performance, security, and reliability is crucial.
We highly recommend reviewing the Best practices for GKE guide, which provides a consolidated overview of best practices for designing, building, and operating applications on GKE.
What’s next?
To start deploying Kimi K3 on Google Cloud, check out one of these paths
-
Deploy Kimi K3 with GB200 on Google Cloud AI Hypercomputer
As Kimi K3 begins to see widespread adoption, we are actively expanding the ecosystem to support more complex serving patterns and next-generation accelerators:
- Distributed Inference with llm-d: For massive-scale serving, standard multi-node configurations face memory-bandwidth and request queuing limits. We are validating recipes for llm-d to support intelligent KV-aware routing, tiered KV cache offloading, Wide EP and disaggregated prefill-decode serving for Kimi K3.
- Next-Generation Hardware Support: Beyond A4 and A4X VMs, we are expanding our validated recipes to support next-generation compute environments, including A4X-Max and Google Cloud TPUs.
Acknowledgements
We’d like to thank Moonshot AI and RadixArk for their close collaboration and partnership to deliver this great outcome for the community.