Can we write an AWS Lambda function to query Kinesis Streams

Question

I have Amazon Kinesis Streams, which has all the click stream data and we want to write an API to query Kinesis Streams.

My plan is to create a API gateway which calls an AWS Lambda function that would query Kinesis Streams and return it back.

Is it possible to use Lambda to query Kinesis Streams or should we use Kinesis Analytics and Lambda?

Answer 1

Your approach is not the typical way to use Amazon Kinesis Streams.

The normal situation is that data is sent to a Kinesis stream, which holds the data for 24 hours (but can be configured for up to 7 days). Your app can then extract the data from the stream and processes it -- either updating something in real-time or storing information in a database. In fact, Amazon Kinesis Firehose can automatically store incoming data into Amazon S3, Amazon Redshift or Amazon Elasticsearch Service.

Thus, streams are processed rather than queried . Just think of them as replayable queues (rather than databases).

Alternatively, you can have Amazon Kinesis Streams trigger an AWS Lambda function whenever a new message arrives in the stream. This allows near-realtime processing of incoming information.

_{(source: amazon.com )}

Answer 2

You can use AWS Kinesis with Lambda via the Stream-based model. AWS Lambda will itself poll the kinesis stream and fetch the number of records(configured by you) from the Kinesis.

Here's more information: http://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html

Answer 3

Use something like Firehose and then send it to Elasticsearch which you can then use to query the data. A Kinesis stream is like a temporary buffer to retain data in until you can do something else with it such as load it into a data store.

See my comment below you can use Kinesis Analytics for what you would like to do.