How can I refresh the aws kinesis iterator when it get expired?

Question

In my spring boot(1.5.7) application I was using the camel kinesis component for reading data from kinesis stream. I have set the Iterator type as "LATEST" as show below

@Override
public void configure() throws Exception {
    from("aws-kinesis://"+stream+"?amazonKinesisClient=#awsClient&iteratorType="LATEST")
            .bean(Settlement.class, "transform")
            .to("log:out");

}

While reading continuously from the kinesis stream there were no issues. But when there were some delay in processing the messages due to errors while processing and if that delay interval is beyond 5 minutes, I was getting an ExpiredIteratorException error as shown below

om.amazonaws.services.kinesis.model.ExpiredIteratorException: Iterator expired. The iterator was created at time Mon Nov 27 15:30:29 UTC 2017 while right now it is Mon Nov 27 15:36:13 UTC 2017 which is further in the future than the tolerated delay of 300000 milliseconds. (Service: AmazonKinesis; Status Code: 400; Error Code: ExpiredIteratorException; Request ID: c655b6d8-58f4-c087-94c3-85f5e7cfdd03)

I have tried using it in camel versions 2.20.1 and 2.20.0. Is there any way to refresh the iterator? or any other better solutions to overcome this problems in such a way I can read the next message available in the stream.

Answer 1

You can use subscribe_to_shard providing ConsumerARN , ShardId , position , sequence number as parameters for again reconnecting to shard as iterator expires in 300000 milliseconds.

[aws doc] : https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html