EC2 上 S3 区域之间的 aws cli 复制

Question

I am trying to copy between two S3 buckets in different regions using the Command Line Interface on an EC2 server.

region info:
EC2 instance: us-west-2
S3 origin: us-east-1
S3 destination: us-west-2

The following commands work perfectly from the EC2 server:
aws s3 cp s3://n-virginia/origin s3://n-virginia/destination --recursive --source-region us-east-1 --region us-east-1 --profile my_profile

aws s3 cp s3://oregon/origin s3://oregon/destination --recursive --source-region us-west-2 --region us-west-2 --profile my_profile

I need to run the following command from the EC2 server:
aws s3 cp s3://n-virginia/origin s3://oregon/destination --recursive --source-region us-east-1 --region us-west-2 --profile my_profile

If I run that command from a local machine it works, but if I run it from the EC2 server that I used for the previous two commands I get the following error:

Error: "A client error (AccessDenied) occurred when calling the CopyObject operation: VPC endpoints do not support cross-region requests"

I am able to copy the files from the origin bucket to the EC2 server, and then copy from the EC2 server to the destination bucket, but this is not an acceptable solution in production. I don't understand why it will work on a local machine but not on the EC2 server ("my_profile" is identical on both machines)

Answer 1

As pointed out in the comments the problem is your VPC has an endpoint and cross region copies are not supported .

To fix that, either temporarily disable the VPC endpoint, by updating your VPC route table, or just create a new VPC without a VPC endpoint and launch an EC2 there.

Cross region replication would be ideal, but as pointed out, that only effects new items in the bucket

Instead of using aws s3 cp you probably want to use aws s3 sync . Sync will only copy changed files, thus allowing you to rerun it again in case it is interrupted. For example:

aws s3 sync s3://n-virginia/origin s3://oregon/destination

Note also that both cp and sync do NOT preserve ACL. So if you have changed ACL permission on individual files they will all be set to the default after the copy. There are some other tools that are supposed to preserve ACL the like https://s3tools.org which seems to work for me.

Answer 2

If downloading the entire bucket locally is not feasible due to disk space required, you can download, upload and remove 5 seconds worth of files.

The first line of shell snippet below starts a background download of the entire source bucket to the local disk. While there are files in the current directory, call aws s3 mv which will copy files to the destination bucket and remove them locally.

mkdir tempdir
aws s3 sync s3://source-bucket . &
sleep 5
while [ $(ls | wc -l) -gt 0 ] ; do mv *.txt tempdir ; aws s3 mv --recursive tempdir/* s3://destination-bucket ; done

The aws s3 sync command creates temporary files, with random extension, while writing files to disk. The aws s3 mv command will unfortunately sometimes upload these files. To avoid this, move a batch of the files, eg all .txt files, to a temporary directory and upload only them.

In practice I see no more than 50M of disk used locally ( less than 500 files where each file is less than 100k)

Answer 3

I know this is an old post but we have faced the same issues recently.

To update the @astrotom response, Amazon S3 Cross-Region Replication (CRR) now supports copying existing objects. you just need to ask for aws support team to unlock the feature. full explanation here and here

From our side, we preferred @brendan solution even though it saturates the network. you can find here a Kubernetes job that can help you automate it.

you can find in this blog multiple approaches to migrate our buckets cross region cross account