How to monitor a AWS S3 bucket with python using boto?

Question

I have access to a S3 bucket. I do not own the bucket. I need to check if new files were added to the bucket, to monitor it.

I saw that buckets can fire events and that it is possible to make use of Amazon's Lambda to monitor and respond to these events. However, I cannot modify the bucket's settings to allow this.

My first idea was to sift through all the files and get the latest one. However, there are a lot of files in that bucket and this approach proved highly inefficient.

Concrete questions:

1. Is there a way to efficiently get the newest file in a bucket?
2. Is there a way to monitor uploads to a bucket using boto?

Less concrete question:

• How would you approach this problem? Say you had to get the newest file in a bucket and print it's name, how would you do it?

Thanks!

Answer 1

You are correct that AWS Lambda can be triggered when objects are added to, or deleted from, an Amazon S3 bucket. It is also possible to send a message to Amazon SNS and Amazon SQS . These settings needs to be configured by somebody who has the necessary permissions on the bucket.

If you have no such permissions, but you have the ability to call GetBucket() , then you can retrieve a list of objects in the bucket. This returns up to 1000 objects per API call.

There is no API call available to "get the newest files".

There is no raw code to "monitor" uploads to a bucket . You would need to write code that lists the content of a bucket and then identifies new objects.

How would I approach this problem? I'd ask the owner of the bucket to add some functionality to trigger Lambda/SNS/SQS, or to provide a feed of files. If this wasn't possible, I'd write my own code that scans the entire bucket and have it execute on some regular schedule.