AWS S3 Bucket indexing using lambda function

Question

I have S3 bucket website which has multiple HTML files, now I have created a lambda function and connected it to S3 trigger. Now I have created a lambda function which I want the first indexAll the current HTML file of S3 bucket to my Elastic search domain indices which I have after that if I upload or delete any HTML file in S3 bucket then this lambda function should index it to ES domain. The issue is that I am not able to index my all current HTML also when I upload a new HTML file I am not able to index them to ES domain indices.

Lambda Function to indexall then index one by one . Also I wan to create a test in AWS lambda test for S3 and use event indexAll to index all file first .

Error is Time out in 3 sec:

Answer 1

Your Lambda function timeout is too low (its default is 3 seconds).

Also, it's not a good idea to try to analyze/index all S3 objects within a single invocation of a Lambda function unless you can constrain the number of objects. Lambda has a maximum timeout of 15 minutes.

One option to deal with existing files, as an alternative to EC2, would be to create a list of existing objects in the bucket (you could just list the bucket if it's reasonably sized, like 10k items or fewer, or you could use an S3 Inventory Report if it's a very large bucket). Either way, get a list of objects and then send them to an SQS queue, one by one. Have SQS trigger your Lambda function one object per invocation or a batch of 10 objects per invocation.