Access objects in S3 using AWS lambda [CDK]

Question

My use case is to trigger a AWS lambda daily, this can be triggered through cloudwatch, every time lambda gets triggered I have to access particular location(prefix) in S3.

This location contains many files and I have to process these files separately. Is this Possible through AWS-CDK?

I tried to check online but I couldn't find any solution to my usecase

PS: Lambda shouldn't be triggered by S3

Answer 1

You can create 2 lambdas with read access of S3 bucket.

Lambda1: will be triggered by CloudWatch/EventBridge event daily. It will read all the files and will push all the object details need to be processed in SNS.

Lambda2 will be triggered by SNS and it will process all the files separately.

Answer 2

Is the particular location(prefix) in S3 fixed or can be changed during the lifetime of the application?

Depending on your use cases, you might want to hard code the prefix in the lambda which is triggered by cloudwatch event, or store the prefix in a database like dynamodb and use lambda to query the database after the lambda is triggered by cloudwatch event.

After that you have 2 options:

1. Use the same lambda which is triggered by cloudwatch event to process the files in S3.

2. Use the lambda which is triggered by cloudwatch event to read the files in S3 and pass the object details to a queue. Another lambda function can be used to query the queue to further process the files. The queue can be SNS or SQS depending on the use case.

Have a look at the architecture here: https://youtu.be/43PhZC3WD8s?t=967 for option 2. Not exactly the same use case, but can be useful!