python 脚本同时删除多个 s3 存储桶的内容并等待它被删除有什么问题？

Question

I am trying to create a python script to delete the contents of 6 s3 buckets simultaneously, wait till all the data gets deleted, and handle more than 1000 objects in a bucket. However, I am randomly encountering the error "KeyError: 'endpoint_resolver'". I have set the AWS configuration correctly as I can list the S3 buckets by running the AWS command. Can you help me resolve this issue?

The code I have written is as follows:

import boto3
import concurrent.futures

def delete_s3_bucket_contents(bucket_name):
    sess = boto3.session.Session()
    s3 = sess.client('s3')
    bucket = boto3.resource('s3').Bucket(bucket_name)
    objects_to_delete = [{'Key': obj.key} for obj in bucket.objects.all()]

    while objects_to_delete:
        response = s3.delete_objects(
            Bucket=bucket_name,
            Delete={
                'Objects': objects_to_delete[:1000],
                'Quiet': True
            }
        )
        objects_to_delete = objects_to_delete[1000:]

def delete_multiple_buckets(bucket_names, max_workers=6):
    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = [executor.submit(delete_s3_bucket_contents, bucket) for bucket in bucket_names]
        concurrent.futures.wait(futures)
        for future in concurrent.futures.as_completed(futures):
            future.result()

bucket_names = ["A","B","C","D","E","F"]
delete_multiple_buckets(bucket_names)

I also tried to delete data from the above 6 buckets simultaneously in bash.

  parallel -j 6 "aws s3api list-objects --bucket {} --query '{Contents: [Contents[].{Key: Key}]}' --output json | jq -r '.Contents[].Key' | xargs -I {} -n 1000 aws s3api delete-objects --bucket {} --delete '{\"Objects\":[{\"Key\":\"{}\"}],\"Quiet\":true}' " ::: "${destination_buckets[@]}"

but it was throwing jq error

jq: error (at <stdin>:189150): Cannot index array with string "Key"

I can run aws rm command but it very slow in deletion

Answer 1

Following the general example , you can try to create session and use it to make the client in your delete_multiple_buckets function and just pass the client to your worker.

import boto3.session
from concurrent.futures import ThreadPoolExecutor

def delete_s3_bucket_contents(client, bucket_name):
    # Put your thread-safe code here

def delete_multiple_buckets(bucket_names, max_workers=6):
    # Create a session and use it to make our client
    session = boto3.session.Session()
    s3_client = session.client('s3')

    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
        # just pass your client as an argument
        futures = [executor.submit(delete_s3_bucket_contents, s3_client, bucket) for bucket in bucket_names]