AWS Lambda create folder in S3 bucket

Question

I have a Lambda that runs when files are uploaded to S3-A bucket and moves those files to another bucket S3-B. The challenge is that I need create a folder inside S3-B bucket with a corresponding date of uploaded files and move the files to the folder. Any help or ideas are greatly apprecited. It might sound confusing so feel free to ask questions.Thank you!

Answer 1

Just to clear up some confusion, in S3 there is no such thing as a folder. What you see in the interface is actually running the ListObjects using a prefix. The prefix is what you are seeing as the folder hierarchy.

To help illustrate this an object might have a key (which is a piece of metadata that defines its name) of folder/subfolder/file.txt , in the console you're actually using a prefix of folder/subfolder/* . This makes sense if you think of S3 more like a key value store, where the value is the object itself.

For this reason you can make a key on a prefix that has not existed before without creating any other hierarchical features.

In your Lambda function, you will need to download the files locally and then upload them to their new object key (remembering to delete the old object). Some SDKS will have an automated function that will perform all of these steps for you (such as Boto3 with the copy function).

Answer 2

Here's a Lambda function that can be triggered by an Amazon S3 Event and move the object to another bucket:

import json
import urllib
from datetime import date
import boto3

DEST_BUCKET = 'bucket-b'

def lambda_handler(event, context):
    
    s3_client = boto3.client('s3')
    
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])


    dest_key = str(date.today()) + '/' + key
    
    s3_client.copy_object(
        Bucket=DEST_BUCKET,
        Key=dest_key,
        CopySource=f'{bucket}/{key}'
        )

The only thing to consider is timezones. The Lambda function runs in UTC and you might be expecting a slightly different date in your timezone, so you might need to adjust the time accordingly.