[英]How to implement exponential retry in kafka properly?
I am a beginner in Kafka.我是卡夫卡的初学者。
I have been trying to implement exponential retry for failed records in Kafka consumer.我一直在尝试对 Kafka 消费者中的失败记录实施指数重试。 After 4 retries the consumer needs to shut down.
重试 4 次后,消费者需要关闭。 Retries should take place after 1 minute then 5 minute then again after 15 minutes and then 30 minutes.
应在 1 分钟后重试,然后 5 分钟后重试,然后在 15 分钟后重试,然后 30 分钟后重试。 After all these attempts, if retry is not successful then I need to shut down consumer.
在所有这些尝试之后,如果重试不成功,那么我需要关闭消费者。 I have done the following to implment it.
我已经完成了以下操作来实现它。 But after 5 minutes(max.poll.interval), consoumer re-balances.
但 5 分钟后(max.poll.interval),消费者重新平衡。 How to complete all retries attempt(5 attempts in case of failure) and after that shut down the consumer?
如何完成所有重试尝试(在失败的情况下尝试 5 次)然后关闭消费者?
ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
backOffPolicy.setInitialInterval(60000);
backOffPolicy.setMultiplier(5);
backOffPolicy.setMaxInterval(900000);
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setRetryPolicy(retryPolicy());
retryTemplate.setBackOffPolicy(backOffPolicy);
return retryTemplate;
}
private RetryPolicy retryPolicy() {
Map<Class<? extends Throwable>, Boolean> exceptionMap = new HashMap<>();
exceptionMap.put(IllegalArgumentException.class, false);
exceptionMap.put(RecoverableDataAccessException.class, true);
SimpleRetryPolicy simpleRetryPolicy = new SimpleRetryPolicy(4, exceptionMap, true);
return simpleRetryPolicy;
}
max.poll.interval
is the upper limit on the time that kafka cluster would wait for a poll call from the consumer before considering it as "dead". max.poll.interval
是 kafka 集群在将其视为“死亡”之前等待来自消费者的轮询调用的时间上限。 Once one of the consumers of a consumer group is considered dead, kafka cluster would trigger a rebalance on that consumer group.一旦消费者组的一个消费者被认为死亡,kafka 集群将触发该消费者组的重新平衡。
In your use case, you want to wait >51mins (retry time + actual processing time) before making a subsequent poll.在您的用例中,您希望在进行后续轮询之前等待 >51 分钟(重试时间 + 实际处理时间)。 So you would need to increase this property to a sufficient value (>51mins) so that the kafka cluster does not assume that the consumer has died.
因此,您需要将此属性增加到足够的值(> 51 分钟),以便 kafka 集群不会假设消费者已经死亡。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.