Hello All,
We are running deepseek:70b model with SSD on g2-standard-96 instance but we get approximately 25 tps in benchmark tests which is impossible. What is the reason?
Model: L4x8
Image: Deep Learning VM for PyTorch 2.4 with CUDA 12.4 M127
Hi @bastus11 ,
Welcome to Google Cloud Community!
There could be several reasons for low throughput in your benchmark tests. Here’s the breakdown and some possibilities:
GPU underutilization - Sometimes, the model’s batch size or data preprocessing pipeline might not be optimized, which can lead to poor GPU utilization.
If the batch size is too small, it could lead to lower throughput. Larger batches generally make better use of the hardware, especially on large models like DeepSeek:70B. However, this is a balancing act, as the batch size must be adjusted to fit the available GPU memory. You may consider experimenting with batch sizes (ex. 16, 32, 64) to identify an optimal batch size for your model. Check the nvidia forum discussion about the batch size that fits GPU memory.
CUDA Version - You’re using CUDA 12.4, which is new. It’s possible that the specific model or libraries you’re using haven’t fully optimized support for this version yet. Trying to use a stable version of CUDA might help.
You can monitor GPU utilization with nvidia-smi or nvidia-smi dmon command to see if your GPUs are being used efficiently.
For your reference, you may refer to these relevant documentations:
- General information about GPUs on Compute Engine.
- About accelerator-optimized machines (G2 series).
- Documentation on the Deep Learning VM images.
- Information about the Deep Learning Containers available on Google Cloud. These are usually the easiest way to get started with the correct drivers.
- Cloud Monitoring for instance performance.
- Google Cloud Profiler for code-level performance analysis.
- CUDA Optimization
If you need further assistance about your Slow TPS, please reach out to our Google Cloud Support team.
Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.