Amazon Kinesis Stream到S3

Question

I have a Javascript application writing real-time data to Kinesis stream. Is there any readily available connector to use S3 as my consumer from Kinesis Stream? Kinesis Firehose is not an option in my case, as there is another application that reads from Streams to do real-time analytics. Same with Lambda (we don't wanna go all-in on AWS). I looked at KCL (KinesisClientLibrary) which exposes IRecordProcessor, that helps me capture every record from Stream. I am in lines of writing my code such that, it captures every record and send it to S3 as an object. I want to know if there is any readily available connector (in Java) or someone has done this already, before jumping in on implementation. Thanks.

Answer 1

Kinesis Firehose can now read data directly from Amazon Kinesis Streams! https://forums.aws.amazon.com/ann.jspa?annID=4904

there is another application that reads from Streams to do real-time analytics. This should be a problem anymore.

Kinesis Streams -> Your App -> Kinesis Firehose -> S3

Please not that Firehose is only available in the US East (N. Virginia), US West (Oregon), and EU (Ireland) regions and cross-region transfers(eg, Streams@ Mumbai -> Firehose@Ireland) are not supported yet.