How to delete all objects from a S3 folder of various sub folder with Specific name which are older than n days using AWS Ruby SDK

Question

I have a requirement to delete files with prefix application.log which are older than 5 days in an S3 folder.

The file is present inside log-bucket/main-shell/apps/app-main-shell-55f79d74fc-4sx6c/helpkit .

Is there a way where we can list and delete files recursively using AWS Ruby SDK?

Answer 1

Rather than writing your own code you can setup AWS3 life cycle with prefix by using RUBY SDK. In life cycle mentioned after 5th day delete data from particular path.

Below are the reference Links to configure S3 cycle and Ruby SDK.

https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/BucketLifecycle.html

https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html

Answer 2

If you want to perform a 1-time clean-up of objects under a single key prefix, you can use the batch operations on the objects collection.

s3 = Aws::S3::Resource.new()

s3.bucket('bucket-name').objects(prefix: 'some/key/prefix/').batch_delete!

This will list objects with the given key prefix and then issue a batch delete for each page of results. The more objects with the given prefix, the more api calls. It should be 2 requests (1x list, 1x batch delete) per 1k objects to delete.

Please note, this is a destructive operation. Make sure you key prefix is correct before you issue the batch delete.

If you want to do this on a frequent basis, then I would use a bucket lifecycle configuration.