Data Transfer to Cloud bucket creates unwanted folders

Google Cloud Compute Infrastructure
1

I’m using data transfer to load files referenced in a tsv file on my own server. The destination is a folder within a bucket. I would expect it to put the retrieved file directly in that folder, but instead it creates a new folders underneath my bucket destination folder.

Example:

TSV file contains:

https://www.mydomain.com/dir1/dir2/filename.doc

Destination:

MyBucket

Folder:

MyFolder

After running the data transfer, it creates under “My Folder” a sub folder called “www.mydomain.com” and under that “dir1” and under that “dir2”.

So instead of accessing my file at:

bucket.storage.googleapis.com/dir1/dir2/filename.doc

The URL ends up being:

bucket.storage.googleapis.com/dir1/dir2/www.rustyzipper.com/dir1/dir2/filename.doc

I’ve looked through the docs & I can’t find anything that explains this behavior or how to fix it.

Help!?

Thanks!

Rob

2

Hi,

This is the intended behavior as the object name is actually “www.rustyzipper.com/dir1/dir2/filename.doc”. Do note that Cloud Storage operates with a flat namespace, which means that folders do not actually exist within Cloud Storage[1]. Futhermore, This section[2] explains the behavior you are experiencing:

“When an object located at http(s)://[HOSTNAME]:[PORT]/[URL_PATH] is transferred to Cloud Storage, the name of the object in Cloud Storage is [HOSTNAME]/[URL_PATH].”

I hope this clears up the confusion.

[1] https://cloud.google.com/storage/docs/folders

[2] https://cloud.google.com/storage-transfer/docs/create-url-list#url-list-format