How to list and read each of the files in specific folder of an S3 bucket using Python Boto3

Question

I have some files in a specific folder of an s3 bucket. All file names are in the same pattern like below:

s3://example_bucket/products/product1/rawmat-2343274absdfj7827d.csv
s3://example_bucket/products/product1/rawmat-7997werewr666ee.csv
s3://example_bucket/products/product1/rawmat-8qwer897hhw776w3.csv
s3://example_bucket/products/product1/rawmat-2364875349873uy68848732.csv
....
....

Here, I think we can say:

bucket_name = 'example_bucket'
prefix = 'products/product1/'
key = 'rawmat-*.csv'

I need to read each of them. I highly prefer not to list of the objects in the bucket.

What could be the most efficient way to do this?

Answer 1

Iterate over the objects at the folder using a prefix

bucket_name = 'example_bucket'
prefix = 'products/product1/rawmat'

for my_object in bucket_name.objects.filter(Prefix= prefix):
    print(my_object)