Unzip file from S3, write to a CSV file and push back to S3

Question

I have built a lambda that collect logs from a EC2 instance and uploads them to a S3 buckets on a daily basis. The logs are stored as .gz files, and now I want to build another lambda that collects the most recently uploaded log file, unzips it, writes it to a CSV file and then pushes it back up to the s3.

I've managed to collect a log file, unzip it and push it back up but I would like some directions how to target the most recent file in the s3 bucket, and how to write it to a CSV before pushing it back up.

I'm using Python for my lambda, and this is how my code looks like right now:

def lambda_handler(event, context):
s3 = boto3.client('s3', use_ssl = False)

s3.upload_fileobj(
    Fileobj = gzip.GzipFile(
        None,
        'rb',
        fileobj = BytesIO(
            s3.get_object(Bucket='bucketName', Key='key')['Body'].read())),
            Bucket ='bucketName',
            Key ='key')

Answer 1

You don't need to worry about querying the latest object in S3. Just use a S3 Event that triggers your Lambda function.

This means that whenever you Lambda in invoked, it will be invoked with the last inserted object on S3, therefore the most recent.