Java AWS S3 SDK2 how to perform async bulk operations

Question

I'd like to perform bulk operations within S3 asynchronously. For some reason, I could not find a way of doing that in the new SDK2 client built by AWS. I currently use the S3AsyncClient 's putObject method for uploading files, but I'd like to do that in batches for performance improvements. The same applies to downloading files. I've read about multipart uploads but I did not understand if that's what I'm looking for, since my files are relatively small and I don't need a multipart upload. I How can I perform a multi-file upload?

Answer 1

You could use S3TransferManager which is the implementation of TransferManager library adjusted for AWS Java SDK v2 . S3TransferManager provides bulk uploads by using multiple threads.

Keep in mind that in case of S3 you are paying for each PUT request. Each file upload will constitute at least one PUT request. Even if you are using something like uploadDirectory from the S3TransferManager , you will still pay for after each file.

Answer 2

I am not aware of a multi-file upload mode in Amazon S3.

Another way is to zip the files and upload one big file, where you will benefit from multipart uploads. Then you let a Lambda function , triggered by the upload of a zip in your bucket, unzip automatically the files for you.

Further description of alternatives and a code example for the zip & unzip strategy can be found here .