Writing to S3 via Kinesis Stream or Firehose

Question

I have events that keep coming which I need to put to S3. I am trying to evaluate if I muse use Kinesis Stream or Firehose. I also want to wait for few minutes before writing to S3 so that the object is fairly full.

Based on my reading of Kinesis Data stream, I have to create an analytics app which will then be used to invoke a lambda. I will then have to use the lambda to write to S3. Or Kinesis Data Streams can directly write to lambda somehow? I could not find anything indicating the same.

Firehose is not charged by hour(while stream is). So is firehose a better option for me?

Answer 1

Or Kinesis Data Streams can directly write to lambda somehow?

Data Streams can't write directly to S3. Instead Firehose can do this:

delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3) , Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), Splunk, and any custom HTTP endpoint or HTTP endpoints owned by supported third-party service providers, including Datadog, MongoDB, and New Relic.

What's more Firehose allows you to buffer the records before writing them to S3. The writing can happen based on buffer size or time. In addition to that you can process the records using lambda function before writing to S3.

Thus, colectively it seems that Firehose is more suited to your use-case then Data Streams.