How can I transfer objects in buckets between two aws accounts with python?

Question

I need to transfer all objects of all buckets, with the same folders and buckets structures, from an aws account to another aws account. I've been doing it with this code through aws cli, one bucket at a time:

aws s3 sync s3://SOURCE-BUCKET-NAME s3://DESTINATION-BUCKET-NAME --no-verify-ssl

Can I do it with python for all objects of all buckets?

Answer 1

The AWS CLI actually is a Python program. It includes multi-threading to copy multiple objects simultaneously, so it will likely be much more efficient than the equivalent Python program you can make yourself.

You can tweak some settings that might help: AWS CLI S3 Configuration — AWS CLI Command Reference

There is no option to copy "all buckets" -- you would still need to sync/copy one bucket at a time.

Another approach would be to use S3 Bucket Replication , where AWS will replicate the buckets for you. This now works on existing objects. See: Replicating existing objects between S3 buckets | AWS Storage Blog

Or, you could use S3 Batch Operations , which can take a manifest (a listing of objects) as input and then copy those objects to a desired destination. See: Performing large-scale batch operations on Amazon S3 objects - Amazon Simple Storage Service

Answer 2

aws s3 sync is a high level functionality not available in AWS SDKs such as boto3 . You have to implement it yourself on top of boto3 or search though many available code snippets that already implement that, such as python - Sync two buckets through boto3 - Stack Overflow .