Python Boto3 - upload images to S3 in one put request

Question

I have a script in python built out, that uploads images to s3, one image|put request at a time. Is it possible to upload all images to s3 at the same time, using one put request, to save $$ on requests?

for image_id in list_of_images:
#upload each image
    filename = id_prefix+"/"+'{0}.jpg'.format(image_id)
    s3.upload_fileobj(buffer, bucket_name, filename, ExtraArgs={ "ContentType": "image/jpeg"})

Answer 1

No.

The Amazon S3 API only allows creation of one object per API call.

Your options are to loop through each file (as you have done) or, if you wish to make it faster, you could use multi-threading to upload multiple files simultaneously to take advantage of more networking bandwidth.

If your desire is simply to reduce requests costs, do not panic. It is only $0.005 per 1000 requests .

Answer 2

Good news - it looks like several companies came out with boto3-like apis, with much better storage + download (per gb) pricing.

As of a few days ago - Backblaze came out with S3 compatible storage + API .

We did a few tests on our application, and everything seems to be working as advertised!