Deleting a large folder from Google Cloud Storage

Question

I have a folder in a Google Cloud Storage bucket, which has millions of files that I need to remove.

What is an efficient way to delete this large folder of files, without having to delete the entire bucket?

I've tried using the gsutil rm command, but it seems like it will take a long time to finish deleting all files.

Furthermore, I also read about Object Lifecycle Management policies, but I read that they apply to the entire bucket, as opposed to any specific folder.

Thanks for your help: :)

Answer 1

gsutil rm will be fastest. Your only alternative is to write code to list and delete each one, which is what gsutil is going to do for you.

Answer 2

Try the Storage Transfer Service , my experience is it can delete 1-1.5k objects per second.

Delete files by copying from a source bucket (or folder in a bucket) that is empty, to a destination bucket or folder that you want to be empty and use the delete files at destination that aren't at the source option.

If using the GUI select this bullet in the advanced transfer options dialog:

You can also create and run the job from the CLI. This example assumes you have access to gs://bucket1/empty/ (which has no objects in it) and you want to delete all objects from gs://bucket2/folder1/ :

gcloud transfer jobs create \
gs://bucket1/empty/ gs://bucket2/folder1/ \
--delete-from=destination-if-unique \
--project my-project

If you want your deletes to happen even faster you'll need to create multiple transfer jobs and have them target different sections of the bucket. Because it has to do a bucket listing to find the files to delete you'd want to make the destination paths non-overlapping (eg gs://bucket2/folder1/ and gs://bucket2/folder2/, etc). Each job will process in parallel at speed getting the job done in less total time.