How can I get the most recent record from an Amazon Kinesis stream?

Question

I would like to get the most recent record from an Amazon Kinesis stream. I intend to extract the timestamp from this record and compare it to the timestamp of the last record checkpointed by a consumer app in order to check whether the consumer is falling behind.

I cannot use the shard iterator type LATEST. This is because LATEST points to just after the most recent record, so it cannot be used to access the most recent record.

Is there a simple way to get the latest record?

An approach I am considering is to get the shard iterator for the sequence number of the record most recently processed by the consumer, make a GetRecords request using that shard iterator, get the next shard iterator from the result of the request, and repeat until a GetRecords request doesn't return any records.

This approach would involve reading all records since the consumer's checkpoint, which seems unnecessarily wasteful. Is there any way around requesting all these records?

Answer 1

You can use the metric: GetRecords.IteratorAgeMilliseconds from CloudWatch metrics for Kinesis. See here: http://docs.aws.amazon.com/kinesis/latest/dev/monitoring_with_cloudwatch.html

Note that this is an average of multiple iterators across multiple shards. If you need to have a more fine grained view of this metric across each one of your shards, you can calculate it as part of the processor.

Remember that there is no latest record in a stream with multiple shards, as each shard is independent in its order. If you have multiple shards you need to check the status of each one of them. It is also a best practice to process each one of the shards in an independent worker, which usually result in different lag for each worker.

Another important note is that you can have multiple processes for a kinesis stream. One can put the events into Redshift and one can process the events to create real time counters in your operational dashboard. It means that you can have a different lag for each type of processor, and not only for each shard.