Unable to mount bucket when execute VertexAI notebook

Google Cloud Build with AI Custom ML & MLOps
,
1

I am unable to execute my VertexAI Instance notebook, because I cannot get gcsfuse to mount my google buckets. The gcsfuse commands work perfectly well in the regular VertexAI notebook, but do not work when I attempt to execute the notebook.

When I attempt to execute the notebook, I choose my service account, which I have ensured has the correct permissions to access the buckets. Furthermore, I have attempted to set the project in the notebook script using

!gcloud config set project {PROJECT_ID}

, in addition to authorising the service account using a key

!gcloud auth activate-service-account --key-file key.json

and enabling the account using

!gcloud iam service-accounts enable

I get the following error ’

{"time":"08/01/2024 05:21:11.288365","severity":"INFO","msg":"Start gcsfuse/1.3.0 (Go version go1.21.4) for app \"\" using mount point: /gcs_raw\n"}
daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithStorageHandle: Mount: mount: running /usr/bin/fusermount: exit status 1

Any help with this would be greatly appreciated!

2

This is the log when I run gcsfuse with debugging

{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899511152272},"severity":"INFO","msg":"Start gcsfuse/1.3.0 (Go version go1.21.4) for app \"\" using mount point: /home/jupyter/trial\n"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899511203236},"severity":"INFO","msg":"Creating Storage handle..."}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899513292480},"severity":"INFO","msg":"Creating a mount at \"/home/jupyter/trial\"\n"}

WARNING: gcsfuse invoked as root. This will cause all files to be owned by
root. If this is not what you intended, invoke gcsfuse as the user that will
be interacting with the file system.
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899513341995},"severity":"INFO","msg":"Creating a new server...\n"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899513360220},"severity":"INFO","msg":"Set up root directory for bucket acoustic-processing-outputs"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899513394032},"severity":"TRACE","msg":"gcs: Req              0x0: <- ListObjects(\"\")"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899569732390},"severity":"TRACE","msg":"gcs: Req              0x0: -> ListObjects(\"\") (56.336536ms): OK"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899569828014},"severity":"INFO","msg":"Mounting file system \"acoustic-processing-outputs\"..."}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899569878290},"severity":"TRACE","msg":"fuse_debug: Beginning the mounting kickoff process"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899569897607},"severity":"TRACE","msg":"fuse_debug: Parsing fuse file descriptor"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899569909249},"severity":"TRACE","msg":"fuse_debug: Preparing for direct mounting"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899569956053},"severity":"TRACE","msg":"fuse_debug: Directmount failed. Trying fallback."}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899570210507},"severity":"TRACE","msg":"fuse_debug: Creating a socket pair"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899570251469},"severity":"TRACE","msg":"fuse_debug: Creating files to wrap the sockets"}
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899570279949},"severity":"TRACE","msg":"fuse_debug: Starting fusermount/os mount"}
/usr/bin/fusermount: fuse device not found, try 'modprobe fuse' first
{"time":{"timestampSeconds":1704796899,"timestampNanos":1704796899571595571},"severity":"ERROR","msg":"Error while mounting gcsfuse: mountWithStorageHandle: Mount: mount: running /usr/bin/fusermount: exit status 1\n"}
mountWithArgs: mountWithStorageHandle: Mount: mount: running /usr/bin/fusermount: exit status 1
​
1 Like
3

Have you managed to solve this issue?

Thanks!

1 Like