GCP Batch use NVDIA GPU to train models what installation are required?

Google Cloud Compute Infrastructure
, , ,
21

Hi @AvishaiW ,

OOC, is there any specific reason you wanted to use COS image instead of Debian image. Batch Debian image provides almost the same as Batch COS image for GPU support. Specifying the boot disk image as batch-debian should already help overcome the issue and use Debian image as Batch job’s default image, similar as the Deep Learning Linux.

If you do want to use the specific Deep Learning image instead of the Batch image, you can also specify the Deep Learning image url on the boot disk field following https://cloud.google.com/batch/docs/specify-vm-os-image.

Hope this helps!

22

This helps, thank you!

23

@wenyhu so what is our conclusion here? We should not use

batch-cos but use batch-debian instead when creating container only gpu jobs?

24

But i tried using batch-debian instead of batch-cos to run container-only job and specified --runtime=nvidia in the options field and i got the error from log: docker: Error response from daemon: unknown or invalid runtime name: nvidia

But from the log it seems that nvidia driver and nvidia container toolkit is installed?

[BATCH NVIDIA Container Toolkit]: NVIDIA Container Toolkit installed

25

@gradientopt no need to specify any custom option when you specify the batch-debian image. It just works.

26

Thanks! @wenyhu I am wondering why batch-cos would not work? Ideally we would want this to work since we are running container jobs and batch-cos should be more efficient?

27

That might be related to your docker image requirement. For most of the GPU container job cases, both Batch COS image and Debian image work. Container-Optimized OS image is a Google only image designed for read-only files, there might be limitations.

You can refer to https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus for more information.

28

Hi there,

Following all your advices here, I tried launching a batch job on a g2 instance, using “batch-debian” image with no option, but when running a check via torch.cuda.is_available() I get a False.

I also tried using the deep learning image directly () but it failed to start with this error message: Failed to reload sshd.service: Unit sshd.service not found.

@wenyhu would you mind helping me notice what I am doing wrong ?

29

Hi there,

I finally got it working with batch-debian as boot disk and –gpus all as unique option, note that adding the devices as mentioned here did not help.