AWS Kinesis Data Firehose 和 Lambda

Question

I have different data sources and I need to publish them to S3 in real-time. I also need to process and validate data before delivering them to S3 buckets. So, I have to use AWS Lambda and validating data. The question is that what is the difference between AWS Kinesis Data Firehose and using AWS Lambda to directly store data into S3 Bucket? Clearly, what is the advantages of using Kinesis Data Firehose? because we can use AWS Lambda to directly put records into S3!

Answer 1

We might want to clarify near real time, as for me, it is below 1 sec.

Kinesis Firehose in this case will batch the items before delivering them into S3. This will result in more items per S3 object. You can configured how often you want the data to be stored. (You can also connect a lambda to firehose, so you can process the data before delivering them to S3). Kinesis Firehose will scale automatically.

Note that each PUT to S3 as a cost associated to it.

If you connect your data source to AWS Lambda, then each event will trigger the lambda (unless you have a batching mechanism in place, which you didn't mention) and for each event, you will make a PUT request to S3. This will result in a lot of small object in S3 and therefore a lot of S3 PUT api. Also, depending on the number of items received per seconds, Lambda might not be able to scale and cost associated will increase.