List AWS S3 folders with boto3

Question

I have boto code that collects S3 sub-folders in levelOne folder:

import boto

s3 = boto.connect_s3()
bucket = s3.get_bucket("MyBucket")

for level2 in bucket.list(prefix="levelOne/", delimiter="/"):
    print(level2.name)

Please help to discover similar functionality in boto3. The code should not iterate through all S3 objects because the bucket has a very big number of objects.

Answer 1

I think the following should be equivalent:

import boto3

s3 = boto3.resource('s3') 

bucket = s3.Bucket('MyBucket')

for object in bucket.objects.filter(Prefix="levelOne/", Delimiter="/"):
    print(object.key)

Answer 2

If you are simply seeking a list of folders, then use CommonPrefixes returned when listing objects. Note that a Delimiter must be specified to obtain the CommonPrefixes :

import boto3

s3_client = boto3.client('s3')

response = s3_client.list_objects_v2(Bucket='BUCKET-NAME', Delimiter = '/')

for prefix in response['CommonPrefixes']:
    print(prefix['Prefix'][:-1])

If your bucket has a HUGE number of folders and objects, you might consider using Amazon S3 Inventory , which can provide a daily or weekly CSV file listing all objects.