blob_list not listing current directory.

Google Cloud Data Analytics
1

Hi,
I have a scenario, tried multiple options but it is not working.
Suppose I have a directory/files,
Source:
gs://bucket_name/table_nm/table_nm_YYYYMMDD/file1.parquet
gs://bucket_name/table_nm/table_nm_YYYYMMDD/file2.parquet
Move files to archive
gs://archive_bucket_name/table_nm/table_nm_YYYYMMDD/file1.parquet
gs://archive_bucket_name/table_nm/table_nm_YYYYMMDD/file2.parquet

I need to copy both the files into a archive path, Code I have works fine to copy and delete file1.parquet and file2.parquet.
After copying files to archive I want delete the delete the folder=table_nm_YYYYMMDD from source but it get retains in source, how do I delete this folder?

Below code just list the file1.parquet and file2.parquet but doesnā€™t list the current folder.

Any suggestion please?

for blob in client.list_blobs(lz_bucket, prefix=lz_folder):
files_blobs.append(blob)

Note: I tried passing lz_folder ending with ā€œ/ā€ and without ā€œ/ā€, still it doesnā€™t work.

Thanks in advance for your help.

2

GCS does not have the concept of folders in the traditional file system sense. The directory structure you see in GCS is actually a convention formed by the object name prefixes. Therefore, deleting a ā€œfolderā€ in GCS means deleting all the objects that share a common prefix.

In your current code, youā€™re listing files within a specific prefix but not addressing the deletion of the folder itself. To effectively delete the folder, you need to delete all objects that have the prefix table_nm_YYYYMMDD/.

Hereā€™s an updated approach to achieve this:

  1. List and Delete All Objects with the Prefix: Use the list_blobs method to list all objects with the specified prefix and then delete them. Ensure that the prefix includes the trailing slash to target all objects within the specific directory.

    from google.cloud import storage

# Initialize the client
client = storage.Client()

# Specify bucket and folder details (replace with your values)
lz_bucket = 'your-bucket-name'
lz_folder = 'table_nm/table_nm_YYYYMMDD'

# Define the prefix with a trailing slash
prefix = f"{lz_folder}/"

# List and delete all objects with the prefix
bucket = client.bucket(lz_bucket)
blobs = client.list_blobs(bucket, prefix=prefix)
for blob in blobs:
    blob.delete()

    ``

    Replace 'your-bucket-name' and 'table_nm/table_nm_YYYYMMDD' with your actual bucket name and folder path. This script will delete all objects within the specified ā€˜folderā€™ in GCS.

  2. Caution: Itā€™s important to use such scripts with caution. Deletion operations are irreversible, and itā€™s crucial to ensure that the script behaves as expected. Always test thoroughly in a safe environment before applying it to your production data.

By following this approach, you effectively delete the ā€œfolderā€ (i.e., all objects with the given prefix) from GCS. Remember, the inclusion of the trailing slash in the prefix is vital to ensure accurate targeting of the objects within the specific directory.

1 Like
3

Thanks for the adviceā€¦ looks like it is working.