I’m trying to run gemma4 12B in google cloud run with Nvidia L4 GPU as per the below steps but get an error :error=“llama-server process has terminated: CUDA error\nCUDA error: device kernel image is invalid: CUDA error\nCUDA error: device kernel image is invalid”
***If i run gemma3:4b everything works fine.
Any ideas how to resolve the CUDA issue.
Following this Guide: Run LLM inference on GPUs with Gemma 4 and Ollama | Google Cloud Documentation
**Google Cloud Run Ollama-Gemma4 **
1. Install services
2. Create a folder ollama-backend and Dockerfile
FROM ollama/ollama:latest
# Listen on all interfaces, port 8080
ENV OLLAMA_HOST 0.0.0.0:8080
# Store model weight files in /models
ENV OLLAMA_MODELS /models
# Reduce logging verbosity
ENV OLLAMA_DEBUG false
# Never unload model weights from the GPU
ENV OLLAMA_KEEP_ALIVE -1
# Store the model weights in the container image
ENV MODEL gemma4:12b
RUN ollama serve & sleep 5 && ollama pull $MODEL
# Start Ollama
ENTRYPOINT [“ollama”, “serve”]
3. Create a bucket in asia-southeast1 as gpu is in the same location
gcloud storage buckets create gs://vllm-gemma4-12b-asia-builds --location=asia-southeast1
4. Build the image
gcloud builds submit \
--(PII Removed by Staff) \
--machine-type e2-highcpu-32 \
--gcs-source-staging-dir=“gs://vllm-gemma4-12b/source”
5.Build the cloud Run container
gcloud beta run deploy ollama-gemma4 \
–(PII Removed by Staff)t \
–concurrency 4 \
–cpu 8 \
–set-env-vars OLLAMA_NUM_PARALLEL=4 \
–gpu 1 \
–gpu-type nvidia-l4 \
–no-gpu-zonal-redundancy \
–max-instances 1 \
–memory 32Gi \
–allow-unauthenticated \
–no-cpu-throttling \
–timeout=600 \
–region asia-southeast1
6. Run the proxy
gcloud run services proxy ollama-gemma4 --port=9090 --region=asia-southeast1
7. Test locally via proxy
curl -s -N localhost:9090/api/generate -d '{
“model”: “gemma4:12b”,
“prompt”: “Why is the sky blue?”
}’ | jq -j ‘.response’