KCL consumer missing records

Question

I observe that there are some records that are present in the kinesis stream but they have not been received by the KCL consumer application. What could be the reasons that this could happen? Records before and after are coming fine.

All exceptions are consumed in my application and are not received by KCL. So the case of http://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html#w1ab1c11c11c15b9 is not happening either.

Some details: number of open shards: 4 number of closed shards: 3 number of workers running: 5

I fetched the sequence number from my Kinesis producer and directly fetched the records using AT_SEQUENCE_NUMBER as the shard iterator type in aws cli and I can see these records.

So the stream has them but the KCL application has not received them.

Answer 1

Look at the value of "millis_behind_latest" in the kinesis response. If the value is ~86399000 & your stream retention period is 24 hours(86400000), then that explains why you are getting an empty response.

By the time you use the shard_iterator to retrieve the record, the record is no longer in the stream as the retention period of the record has been exceeded. Hence you get an empty result because the oldest record has expired and no longer there in the data stream. So the shard_iterator is now pointing to an empty space in the disk.

When such a thing happens take the value of "next_shard_iterator" and use get_records to once again get the kinesis data records. Maybe data is not stored in concurrent/contiguous memory memory blocks and hence we get empty results in between retrieval of data.

Keep taking the value of "next_shard_iterator" and use get_records until you get a value of 0 for "millis_behind_latest".

You will definitely get all the data added into the kinesis stream.

Hope this answer helps. :)