Can Python delete specific multiple files in S3?
I want to delete multiple files with specific extensions.
This script removes all files.
These are the various specific files that I want to delete:
XXX.tar.gz
XXX.txt
** Current code: ** (all files deleted)
import boto3
accesskey = "123"
secretkey = "123"
region = "ap-northeast-1"
s3 = boto3.resource ('s3', aws_access_key_id = accesskey, aws_secret_access_key = secretkey, region_name = region)
bucket = s3.Bucket ('test')
files = [os.key for os in bucket.objects.filter (Prefix = "myfolder / test /")]
tar_files = [file to file in files if file.endswith ('tar.gz')]
#print (f'All files: {files} ')
#print (f'CSV files: {csv_files} ')
objects_to_delete = s3.meta.client.list_objects (Bucket = "test", Prefix = "myfolder / test /")
delete_keys = {'Objects': []}
delete_keys ['Objects'] = [{'Key': tar_files} for tar_files in [obj ['Key'] for obj in objects_to_delete.get ('Content', [])]]
s3.meta.client.delete_objects (Bucket = "test", Delete = delete_keys)
If anyone knows, please let me know.
Presuming that you want to delete *.tar.gz
and *.txt
files from the given bucket and prefix, this would work:
import boto3
s3_resource = boto3.resource('s3')
bucket = s3_resource.Bucket('my-bucket')
objects = bucket.objects.filter(Prefix = 'myfolder/')
objects_to_delete = [{'Key': o.key} for o in objects if o.key.endswith('.tar.gz') or o.key.endswith('.txt')]
if len(objects_to_delete):
s3_resource.meta.client.delete_objects(Bucket='my-bucket', Delete={'Objects': objects_to_delete})
import boto3
s3 = boto3.resource('s3')
for bucket in s3.meta.client.list_buckets()['Buckets']:
for count, obj in enumerate(s3.Bucket(bucket['Name']).objects.filter()):
if obj.key.endswith('.tar.gz') or obj.key.endswith('.txt'):
print("{}: deleting: {} from: {}".format(count, obj.key, bucket['Name']))
s3.meta.client.delete_object(Bucket=bucket['Name'], Key=obj.key)
I used this method since, for large buckets creating the complete list and then deleting it can take some time. This way, you see the progress.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.