How to copy all files from one folder to another folder in the same S3 bucket

Question

I am trying to copy all files from one folder to another folder in the s3 bucket. I see a lot of examples for moving from one bucket to another but not from one folder to another.

from the examples I have seen online, I have written the following code.

import boto3
s3_resource = boto3.resource('s3')
src = 'old_folder'
dest = 'new_folder'

for key in s3_resource.list_objects(Bucket=config.S3_BUCKET)['Contents']:
    files = key['Key']
    copy_source = {'Bucket': config.S3_BUCKET, 'Key': f'{src}/{files}'}
    s3_resource.meta.client.copy(
        copy_source, f'{config.S3_BUCKET}/{dest}/', f'{src}/{files}')

however, when I run the code, I get the following error:

AttributeError: 's3.ServiceResource' object has no attribute 'list_objects'

for the following line: for key in s3_resource.list_objects(Bucket=config.S3_BUCKET)['Contents']:

I believe I am getting the error because I am using boto3.resource(s3) instead of boto3.client(s3) however the examples I have seen online seem to be using boto3.resource(s3) to move files from one bucket to another.

What is the correct approach to take to move all files from one folder to another folder in the same bucket in s3 using python?

Answer 1

I modified the code and also added pagination. Please have a look at it:

s3 = boto3.client('s3')

src = 'old_folder'
dest = 'new_folder'

paginator = s3.get_paginator('list_objects')

operation_parameters = {'Bucket': config.S3_BUCKET,
                        'Prefix': src}

page_iterator = paginator.paginate(**operation_parameters)

for page in page_iterator:
    for obj in page['Contents']:
        file = obj['Key']
        #print(file)    
        dest_key = file.replace(src, dest)
        #print("dest_key)
        s3.copy_object(Bucket=config.S3_BUCKET, 
                       CopySource=f'/{config.S3_BUCKET}/{file}', 
                       Key=dest_key)