What is the cheapest way to transfer files from one S3 bucket to another?

Question

I was going to straight up copy paste everything via a Commander One file manager, but turns out that's going to cost a fortune. Cuz I guess that counts as downloading and then reuploading everything. I was told to use an instance instead and transfer them that way (still trying to figure out how to do that). But are there any other simple(r) ways maybe?

trying to attach this policy to the destination bukkit:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {"AWS":"arn:aws:iam::<id number here>:user/<username here>"},
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket to get files from>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Principal": {"AWS":"arn:aws:iam::<id number here>:user/<username here>"},
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket to put files in>"
            ]
        }
    ]
}

getting policy has invalid resource error

Answer 1

The easiest way to do it is using sync, it's pretty straight forward:

aws s3 sync s3://sourcebucket s3://destinationbucket

Read more at: https://aws.amazon.com/premiumsupport/knowledge-center/copy-s3-objects-account/

You also do have to adjust the bucket policies of the source and destination bucket to allow gets & puts respectively.

As for cost, if its in the same region, it's free. You are technically not transferring data out to the internet, you are transferring it to another s3 bucket within the same region.

Answer 2

You should use the CopyObject() command, which copies objects directly between Amazon S3 buckets with no need to download/upload. Since the buckets are in the same region, the only cost is a few API request ($0.005 per 1,000 requests).

The easiest way to do this is with the AWS Command-Line Interface (CLI) , using either the aws s3 cp or aws s3 sync command. It will call CopyObject() on your behalf. For example:

aws s3 sync s3://bucket-a/folder/ s3://bucket-b/folder/

Please note that the credentials you use to perform the copy must have read permission on the source bucket and write permission on the destination bucket.

Let's say you are copying from Bucket-A in Account-A to Bucket-B in Account-B . This can be done either by:

• Using credentials from an IAM User ( User-B ) from Account-B . Add a bucket policy on Bucket-A allowing User-B to read from the bucket, OR
• Using credentials from an IAM User ( User-A ) from Account-A . Add a bucket policy on Bucket-A allowing User-A to write to the bucket. When copying, be sure to add --acl bucket-owner-full-control to grant object ownership to Account-B .

I prefer the first option, since the destination account is 'pulling' the files into its own account, so it owns the files by default.