Upload multiple files to S3 using Lambda/Python

Question

so I'm writing a Lambda function that is triggered by events from DynamoDB Streams, and I want to write these events to S3 (to create a data lake). but this code is only uploading the same json file. How can I upload multiple files into s3 without overwriting this one?

   import boto3
   import json
   
   s3 = boto3.client('s3')
   
   def lambda_handler(event, context):
       
     bucket ='bto-history'
     dynamodb = boto3.resource('dynamodb')
     tableUsers = dynamodb.Table('Users')
       
     jsonToUpload = event['Records']
   
     uploadFile = bytes(json.dumps(jsonToUpload).encode('UTF-8'))
     
     jsonToUpload = "userUpdate" + ".json"
   
     s3.put_object(Bucket=bucket, Key=jsonToUpload, Body=uploadFile)
   
     
     return {
       'statusCode': 200,
       'body': event
       }

Answer 1

You didn't say, but I'm guessing that you are writing a Lambda function that is triggered by events from DynamoDB Streams, and you want to write those events to S3.

If you want to maintain multiple files/objects in S3 then you need to give them unique keys. Writing to userUpdate.json will simply overwrite any existing object with that key (unless you have versioning enabled, which I presume you don't).

So, create a unique object key each time. You could insert a timestamp in milliseconds (or other), which is probably unique. Or you could insert a UUID.

Worth asking: why do you want to store the DynamoDB Streams events in S3?

Answer 2

event['Records'] is a list which you need to iterate over. See https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html

Each entry holds info about a dynamodb event.