Download a file from a folder inside S3 bucket in python

Question

I have a bucket name stored as string

s3_dest1 = "s3://fbg-hg/AGLUE/MYSQL/QUERY1/"
s3_dest2 = "s3://fbg-hg/AGLUE/MYSQL/QUERY2/"
s3_dest3 = "s3://fbg-hg/AGLUE/MYSQL/QUERY3/"
s3_dest4 = "s3://fbg-hg/AGLUE/MYSQL/QUERY4/"
s3_dest5 = "s3://fbg-hg/AGLUE/MYSQL/QUERY5/"
s3_dest6 = "s3://fbg-hg/AGLUE/MYSQL/QUERY6/"

I want to download the files from this s3 bucket and end an email as an attachment. There will be only one file in this folder but to get this file we need to iterate over the folder because i will not be knowing the name of the file.

This is what i am doing but this code gives me error.

AttributeError: 'str' object has no attribute 'objects'

Here is my python code

my_list = [s3_dest1, s3_dest2,s3_dest3,s3_dest4,s3_dest5,s3_dest6]
for s3_dest in my_list:
    s3=boto3.client('s3')
    for s3_object in s3_dest.objects.all():
        filename = os.path.split(s3_object.key)
        print(filename)

I am new to python

Answer 1

As is described in this answer and that answer, there are multiple ways to solve this. In your case, you saved the destination as a string, but never did anything with it. You never passed it to boto3 . If I try to get your code to work, it becomes something like

# Use the bucket name, not a connection string
my_list = ['fbg-hg/AGLUE/MYSQL/QUERY1', ...]
attachments = []

s3=boto3.client('s3')
for bucket_name in my_list:
    bucket = s3.Bucket(bucket_name)
    for s3_object in bucket.objects.all():
        filename = os.path.split(s3_object.key)
        print(filename)

        # and then when you want to add it as an attachment
        bytes_buffer = io.BytesIO()
        client.download_fileobj(Bucket=bucket_name,
                                Key=object_key,
                                Fileobj=bytes_buffer)
        byte_value = bytes_buffer.getvalue()
        attachments.append(byte_value.decode())