transfer 1000 files at a time from one s3 bucket to another using python

Question

I want to copy files from one s3 bucket to another. But the thing is I just want to transfer 1000 files at a time and in total I have 300,000 files. And another thing is I want to give 15 sec time between batches to transfer.

I have written below code, which copy from one s3 to another but server is getting stuck. So please help me with this problem.

aws s3 cp s3://sources3bucket/ s3://dests3bucket --recursive

Thanks in advance!

Answer 1

You can use AWS Step Functions to copy files from one Bucket to Another. The steps involves

• Write the Lambda Functions to Copy files from Source based on the given Source Bucket Parameter and to Save it to a Destination Bucket.
• Setup a Step Function Workflow
• You can also add a delay of 15 seconds configured in the Workflow.

For more details refer the Blog Article in AWS Compute Blob Synchronizing Amazon S3 Buckets Using AWS Step Functions .

Sample code is available in SyncBucketsStateMachine Github Repository.