How to find "folder only" in S3 Bucket using S3API or any other utils?

Question

How to get list of "only" Folders (List unknown at this time) in a S3 bucket that has over 80 million objects/keys?

Vendor initially uploaded several millions gz files directly on the root of the bucket (Bucket-Vend) instead of creating folders. When they realized, they then created folder under Bucket-Vend; however did not remove the millions of individual files. To cleanup the mess, we are looking to see if we can get "just" the folder names directly under this bucket (Bucket-Vend). We have tried using Cloudwatch and S3api but unfortunately we are not able to get a list of just Folders at this bucket level. The idea is once we know the folders, we could then move the folders to another bucket and then delete all the individual files and finally the bucket (Bucket-Vend). Any help is appreciated please!

Answer 1

Given the number of objects in your bucket, I would recommend using Amazon S3 Inventory , to obtain a daily CSV listing all objects in the bucket.

The folders might appear as zero-length objects, or you could just extract the folder names from the object names (eg folder1/foo.txt is in folder1 ).

Given the list, you could then write a program to copy the objects to their desired location, or use the AWS Command-Line Interface (CLI) aws s3 mv command that will both "move" the objects (by doing a copy and delete). It can also "move" a folder between buckets.