python boto3 for AWS - S3 Bucket Sync 优化

Question

currently I am trying to compare two S3 buckets with the target to delete files.

problem defintion: -BucketA -BucketB

The script is looking for files (same key name) in BucketB which are not available in BucketA. That files which are only available in BucketB have to be deleted. The buckets contain about 3-4 Million files each.

Many thanks.

Kind regards, Alexander

My solution idea:

The List filling is quite slow. Is there any possibility to accelerate it?

#Filling the lists
#e.g. BucketA (BucketB same procedure)
s3_client = boto3.client(3")
bucket_name = "BucketA"
paginator = s3_client.get_paginator("list_objects_v2")
response = paginator.paginate(Bucket="BucketA", PaginationConfig={"PageSize": 2})
for page in response:
    files = page.get("Contents")
    for file in files:
        ListA.append(file['Key'])

#finding the deletion values
diff = list(set(BucketB) - set(BucketA))
 

#delete files from BucketB (building junks, since with delete_objects_from_bucket max. 1000 objects at once)
for delete_list in group_elements(diff , 1000):
    delete_objects_from_bucket(delete_list)

Answer 1

The ListBucket() API call only returns 1000 objects at a time, so listing buckets with 100,000+ objects is very slow and best avoided. You have 3-4 million objects, so definitely avoid listing them!

Instead, use Amazon S3 Inventory , which can provide a daily or weekly CSV file listing all objects in a bucket. Activate it for both buckets, and then use the provided CSV files as input to make the comparison.

You can also use the CSV file generated by Amazon S3 Inventory as a manifest file for Amazon S3 Batch Operations. So, your code could generate a file that lists only the objects that you would like deleted, then get S3 Batch Operations to process those deletions.