如何使用 boto3 在 2 个不同帐户的 S3 存储桶之间复制文件

Question

I'm trying to files from a vendors S3 bucket to my S3 bucket using boto3. I'm using the sts service to assume a role to access the vendor s3 bucket. I'm able to connect to the vendor bucket and get a listing of the bucket. I run into CopyObject operation: Access Denied error when copying to my bucket. Here is my script

session = boto3.session.Session(profile_name="s3_transfer")
sts_client = session.client("sts", verify=False)
assumed_role_object = sts_client.assume_role(
    RoleArn="arn:aws:iam::<accountid>:role/assumedrole",
    RoleSessionName="transfer_session",
    ExternalId="<ID>",
    DurationSeconds=18000,
)

creds = assumed_role_object["Credentials"]
src_s3 = boto3.client(
    "s3",
    aws_access_key_id=creds["AccessKeyId"],
    aws_secret_access_key=creds["SecretAccessKey"],
    aws_session_token=creds["SessionToken"],
    verify=False,
)
paginator =src_s3.get_paginator("list_objects_v2")
# testing with just 2 items.
# TODO: Remove MaxItems once script works.
pages = paginator.paginate(
    Bucket="ven_bucket", Prefix="client", PaginationConfig={"MaxItems": 2, "PageSize": 1000}
)
dest_s3 = session.client("s3", verify=False)
for page in pages:
    for obj in page["Contents"]:
        src_key = obj["Key"]
        des_key = dest_prefix + src_key[len(src_prefix) :]
        src = {"Bucket": "ven_bucket", "Key": src_key}
        print(src)
        print(des_key)
        dest_s3.copy(src, "my-bucket", des_key, SourceClient=src_s3)

The line dest_s3.copy... is where I get the error. I have the following policy of my aws user to allow copy to my bucket

{
    "Version": "2012-10-17",
   "Statement": [
    {
        "Sid": "VisualEditor1",
        "Effect": "Allow",
        "Action": [
            "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::my-bucket/*",
            "arn:aws:s3:::my-bucket/"
        ]
    }
    ]
}

I get the following error when running the above script.

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

Answer 1

The CopyObject() command can be used to copy objects between buckets without having to upload/download. Basically, the two S3 buckets communicate with each other and transfer the data.

This command can also be used to copy between buckets that in different regions and different AWS accounts.

If you wish to copy between buckets that belong to different AWS accounts , then you will need to use a single set of credentials that have:

• GetObject permission on the source bucket
• PutObject permission on the destination bucket

Also, please note that the CopyObject() command is sent to the destination account . The destination bucket effectively pulls the objects from the source bucket .

From your description, your code is assuming a role from the other account to gain read permission on the source bucket. Unfortunately, this is not sufficient for the CopyObject() command because the command must be sent to the destination bucket. (Yes, it is a little hard to discern this from the documentation. That is why the source bucket is specifically named, rather than the destination bucket.)

Therefore, in your situation, to be able to copy the objects, you will need to use a set of credentials from Account-B (the destination) that also has permission to read from Bucket-A (the source). This will require the vendor to modify the Bucket Policy associated with Bucket-A .

If they do not wish to do this, then your only option is to download the objects using the assumed role and then separately upload the files to your own bucket using credentials from your own Account-B .

Answer 2

I know this is old, but your code works just fine and there's no need to modify it. If you have the destination account (A) and the source account (B) then you correctly invoke Lambda in A to assume a role in B and copy files from S3 in B to S3 in A, just like you did. The core and actual solution to your problem is the proper configuration of permissions. Please, refer to these two AWS docs pages that actually solve the problem:

https://aws.amazon.com/premiumsupport/knowledge-center/copy-s3-objects-account/ https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-assume-iam-role/

One additional thing that I'd add though, is the ACL. This will ensure that objects copied from B to A will be owned by the destination bucket's owner.

dest_s3.copy(
            CopySource=...,
            Bucket=...,
            Key=...,
            ExtraArgs={
                "ACL": "bucket-owner-full-control"
            }
        )

Answer 3

1. On source AWS account, add this policy to the source S3 bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::SOURCE_BUCKET_NAME",
                "arn:aws:s3:::SOURCE_BUCKET_NAME/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::DESTINATION_BUCKET_NAME",
                "arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
            ]
        }
    ]
}

2. Using the destination account's credentials:

boto3_session = boto3.Session(aws_access_key_id=<your access key>,
                              aws_secret_access_key=<your secret_access_key>)
s3_resource = boto3_session.resource('s3')
bucket = s3_resource.Bucket("<source bucket name>")

for obj in bucket.objects.all():
    obj_path = str(obj.key)

    copy_source = {
        'Bucket': "<source bucket name>",
        'Key': obj_path
    }
    s3_resource.meta.client.copy(copy_source, "<destination bucket name>", obj_path)