Delete all keys in a bucket of s3 which are three days old using python

Question

How do I delete all the keys in an S3 bucket which are three days old using python?

S3 bucket contents are like:

mybucket001/backup/1566394660_21_08_2019_backup
mybucket001/backup/1566394660_20_08_2019_backup
mybucket001/backup/1566394660_19_08_2019_backup
mybucket001/backup/1566394660_18_08_2019_backup

I need to keep only the last two days of data.

Here is what I tried:

import boto

from boto.s3.key import Key

keyId='***'
sKeyId='***'

srcFileName="file name" #Name of the file to be deleted

bucketName="bucket name" #Name of the bucket, where the file resides

conn = boto.connect_s3(keyId,sKeyId) #Connect to S3

bucket = conn.get_bucket(bucketName) #Get the bucket object

k = Key(bucket,srcFileName) #Get the key of the given object

k.delete()

Answer 1

您可以简单地将Amazon S3 对象生命周期管理配置为在 3 天后删除对象，而不是通过代码执行此操作。

Answer 2

This seems like a slightly odd way to use boto and S3; I would set up an S3 client:

 import boto3
 import datetime
 s3 = boto3.client('s3')

then use the boto API to list the files in the bucket (assuming the bucket exists):

files = s3.list_objects_v2(Bucket='my-bucket')['Contents']

which will give you a list of dictionaries, each corresponding to one object/file.

You can then filter this list by modified date and extract the keys:

old_files = [{'Key': file['Key']} for file in files if file['LastModified'] < datetime.now() - timedelta(days=2)]

and then again use the API to delete a certain portion of those files:

 s3.delete_objects(Bucket='my-bucket', Delete={'Objects': old_files}) 

Answer 3

Try something like this:

import boto
from datetime import datetime
from boto.s3.key import Key

keyId='***'
sKeyId='***'

srcFileName="file name" #Name of the file to be deleted

bucketName="bucket name" #Name of the bucket, where the file resides

conn = boto.connect_s3(keyId,sKeyId) #Connect to S3

bucket = conn.get_bucket(bucketName) #Get the bucket object

k = Key(bucket,srcFileName) #Get the key of the given object
today = datetime.now().day  # Get current day
if today - int(k.split('_')[1]) >= 2:  # check the day difference for the last 2 days
    k.delete()

run for every key

Answer 4

You don't need to worry at all about the deletion as S3 itself will take that headache for you.

Ref https://aws.amazon.com/.../aws/amazon-s3-object-expiration/