Adding mount_options when mounting a GCS bucket in Batch

Google Cloud Compute Infrastructure
1

Hi there,

We are trying to mount a GCS bucket, but would like to add gcsfuse parameters:

# Jobs can be divided into tasks. In this case, we have only one task.
    task = batch_v1.TaskSpec(
        runnables=[runnable],
        compute_resource=resources,
        max_retry_count=10,
        max_run_duration="36_000s",
        volumes=[
            batch_v1.Volume(
                gcs= batch_v1.GCS(
                    remote_path="XXX/images/",
                ),
                mount_path="/mnt/disks/share",
                mount_options=["max-conns-per-host=200", "max-idle-conns-per-host=200"],
            )
        ],
    )

It works when we remove that argument, but not with it:

mount_options=["max-conns-per-host=200", "max-idle-conns-per-host=200"]

And it’s a bit tricky to debug because when an instance fails when mounting a volume, Batch doesn’t show any log.

I guess the question is: am I passing the options correctly? Is it the correct syntax? Examples in docs don’t show any example passing options with GCS.

Here is where I found the options names: github

1 Like
2

Hi Nlassaux,

You can find the mount options here: https://cloud.google.com/storage/docs/gcsfuse-cli.

For

mount_options=["max-conns-per-host=200", "max-idle-conns-per-host=200"]

you can specify them as

mount_options=["--max-conns-per-host 200", "--max-idle-conns-per-host 200"]

You can follow this guide to set up and analyze logs.

Please let us know if this works for you. :slightly_smiling_face:

Best,

Wen

3

Thank you, that’s super useful! I couldn’t manage to find an example in the documentation, it might be a good thing to add or make more visible.

4

Thanks for the advise Nlassaux! We have a github repo for samples: https://github.com/GoogleCloudPlatform/batch-samples/tree/main. I have a PR for GCS samples waiting for review, hope it could help.

Best,

Wen

1 Like