How to get s3 metadata for all keys in a bucket via boto3

Question

I want to fetch all metadata for a bucket with a prefix via Boto. There are a few SO questions that imply this isn't possible via the AWS API . So, two questions:

1. Is there a good reason this shouldn't be possible via the AWS API?
2. Although I can't find one in docs , Is there a convenience method for this in Boto?

I'm currently doing this using multithreading, but that seems like overkill, and I'd really rather avoid it if at all possible.

Answer 1

While there isn't a way to do this directly through boto, you could add an inventory configuration on the bucket(s) which generates a daily CSV / ORC file with all file metadata.

Once this has been generated you can then process the output rather than multithreading or any other method that requires a huge number of requests.

See: put_bucket_inventory_configuration

Its worth noting that it can take upto 48 hours for the first one to be generated.