how to copy files and folders from one S3 bucket to another S3 using python boto3

Question

I want to copy a files and folders from one s3 bucket to another. I am unable to find a solution by reading the docs. Only able to copy files but not folders from s3 bucket. Here is my code:

import boto3
s3 = boto3.resource('s3')
copy_source = {
    'Bucket': 'mybucket',
    'Key': 'mykey'
 }
s3.meta.client.copy(copy_source, 'otherbucket', 'otherkey')

Answer 1

S3 does not have any concept of folder/directories. It follows a flat structure. For example it seems, On UI you see 2 files inside test_folder with named file1.txt and file2.txt, but actually two files will have key as "test_folder/file1.txt" and "test_folder/file2.txt". Each file is stored with this naming convention.

You can use code snippet given below to copy each key to some other bucket.

import boto3
s3_client = boto3.client('s3')
resp = s3_client.list_objects_v2(Bucket='mybucket')
keys = []
for obj in resp['Contents']:
    keys.append(obj['Key'])



s3_resource = boto3.resource('s3')
for key in keys:
    copy_source = {
        'Bucket': 'mybucket',
        'Key': key
    }
    bucket = s3_resource.Bucket('otherbucket')
    bucket.copy(copy_source, 'otherkey')

If your source bucket contains many keys, and this is a one time activity, then I suggest you to checkout this link .

If this needs to be done for every insert event on your bucket and you need to copy that to another bucket, you can checkout this approach .

Answer 2

S3 is flat object storage, there are no "folders" there. It is structured by key prefix for display purposes in S3 browser. To copy "folder" you have to copy all objects with common key prefix.