S3 对象从一个存储桶复制到另一个存储桶 - 出现拒绝访问错误

Question

I'm trying to copy the data from one s3 bucket to another cross account cross region (from us-west-2 to us-east-1) but keep getting error no matter what s3 copying method I use. Can someone please review this and let me know what am I doing wrong?

An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

Listing the print statement results from the below code here -

Copying file1.json
{SOURCE_BUCKET}/{obj.key}---->demo-bucket-a/year=2022/month=12/day=21/file1.json
cp_source---->{'Bucket': 'demo-bucket-a', 'Key': 'year=2022/month=12/day=21/file1.json'}
TARGET_BUCKET----->demo-bucket-b
obj.key----->year=2022/month=12/day=21/file1.json

import json
import boto3
import urllib
import uuid
import os
import logging
from datetime import datetime, timedelta

logger = logging.getLogger()
logger.setLevel(logging.INFO)
# boto3.set_stream_logger('', logging.DEBUG)

# lambda function to copy file from 1 s3 to another s3
def lambda_handler(event, context):
    
    dt_day = ''
    dt_month = ''
    dt_year = ''
    
    execution_time = datetime.now() - timedelta(3)
    logger.info(f'Start lambda: {execution_time}')
    year = execution_time.year
    month = execution_time.month
    day = execution_time.day
    
    SOURCE_BUCKET = 'demo-bucket-a'
    SOURCE_PATH = f"year={year}/month={month:02}/day={day:02}/"
    

    # Target Bucket Details 
    TARGET_BUCKET = 'demo-bucket-b'
    TARGET_PATH = f"sub-folder-b/"
    # TARGET_LOC = f"year={year}/month={month:02}/day={day:02}/"
    
    kd = 'Detail/'
    ko = 'Owned/'
    ks = 'Summary/'
    
    s3_client = boto3.client('s3')
    s3_resource = boto3.resource('s3')

    bucket = s3_resource.Bucket(SOURCE_BUCKET)
    
    fname = ''
    # For loop prints the file names located in the S3 bucket and source path
    for obj in s3_resource.Bucket(name=SOURCE_BUCKET).objects.filter(Prefix=SOURCE_PATH):
        filename = obj.key.split('/')[-1]
        print(filename)
        
        target_key = obj.key[obj.key.rfind('/')+1:]
        
        if 'summary' in filename:
            fname = ks
        elif 'Owned' in filename: 
            fname = ko
        else:
            fname = kd
        
        cp_source = {'Bucket':SOURCE_BUCKET, 'Key': obj.key}
        
        print('Copying', target_key)
        print("{SOURCE_BUCKET}/{obj.key}---->" + f"{SOURCE_BUCKET}/{obj.key}")
        print("cp_source---->" + str(cp_source))
        print("TARGET_BUCKET----->" + str(TARGET_BUCKET))
        print("obj.key----->" + str(obj.key))
        
        # ACCESS DENIED ERROR for below code -------->
        # s3_resource.Object(TARGET_BUCKET, TARGET_PATH + fname + SOURCE_PATH + target_key).copy({'Bucket':SOURCE_BUCKET, 'Key': obj.key})
        
        # ACCESS DENIED ERROR for below code -------->
        # s3_resource.Object(TARGET_BUCKET, TARGET_PATH + fname + SOURCE_PATH + target_key).copy_from(CopySource=f"{SOURCE_BUCKET}/{obj.key}")
        
        # ACCESS DENIED ERROR for below code -------->
        # s3_client.copy_object(CopySource=cp_source, Bucket=TARGET_BUCKET, Key=obj.key)
        
        s3_resource.meta.client.copy(cp_source, TARGET_BUCKET, f"fname/{obj.key}")
    
    s3_resource.Bucket(name=TARGET_BUCKET).put_object(Body='', Key=TARGET_PATH + 'Manifest/' + SOURCE_PATH + str(uuid.uuid4()) + '.manifest', ACL = 'bucket-owner-full-control')
    
    return {
        'statusCode': 200,
        'body': json.dumps('Executed S3-to-S3-Copy-Lambda Successfully !!!')
    }

I do have the correct access permissions on demo-bucket-a as well as IAM-role-A and demo-bucket-b

Source Bucket demo-bucket-a Permissions/Bucket level policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::xxxxxxxxx:role/iam-s3-copy-lambda-role"
                ]
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket-a",
                "arn:aws:s3:::demo-bucket-a/*"
            ]
        }
    ]
}

Destination IAM Lambda Role 'iam-s3-copy-lambda-role` Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "kms:Decrypt",
            "Resource": "arn:aws:kms:us-east-1:xxxxxxxxx:key/aws-kms-key",
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:GenerateDataKey*",
                "kms:Decrypt"
            ],
            "Resource": "arn:aws:kms:us-east-1:xxxxxxxxx:key/cmk-kms-key",
            "Effect": "Allow"
        },
        {
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetBucketAcl",
                "s3:GetBucketLocation",
                "s3:GetBucketNotification",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectTagging",
                "s3:ListBucket",
                "s3:CopyObject",
                "s3:HeadObject",
                "s3:List*",
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:PutObjectAcl",
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket-b",
                "arn:aws:s3:::demo-bucket-b/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectTagging",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket-a",
                "arn:aws:s3:::demo-bucket-a/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Destination Bucket demo-bucket-b Permissions/Bucket level policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::xxxxxxxxx:role/iam-s3-copy-lambda-role"
            },
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectTagging",
                "s3:PutObjectVersionAcl"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket-b",
                "arn:aws:s3:::demo-bucket-b/*"
            ]
        }
    ]
}

Answer 1

I wanted to provide an alternate solution to you which is Cross-account IAM roles. I find it easier to use Cross-account IAM roles instead of setting up Role and Resource policies in multiple accounts.

The main idea in Cross-account IAM roles is to create a role that can do your tasks (like access to S3) and assume that role from a different account to be able to do the same tasks.

This AWS doc explains it in detail: https://repost.aws/knowledge-center/cross-account-access-s3

Extra reference: https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html