[英]kafka restabilizing group using spring kafka consumer
I have a quite slow consumer which could take more than 5 minutes to process the record. 我的用户使用速度很慢,可能需要5分钟以上才能处理记录。 What I want to avoid is kafka restabilizing the group.
我要避免的是让卡夫卡重新建立团队。 In order to do this from my understanding I have to set to the kafka broker the following properties:
为了实现此目的,我必须将以下属性设置为kafka经纪人:
group.max.session.timeout.ms = 3600001
group.min.session.timeout.ms = 3600000
In my application side I have the following configuration : 在我的应用程序端,我有以下配置:
@Bean
public Map<String, Object> consumerConfigs() {
final Map<String, Object> propsMap = new HashMap<>();
propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
environment.getProperty("app.kafkaBrokers"));
propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
propsMap.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, Integer.valueOf(environment.getProperty("app.session.timeout.ms")) );
propsMap.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, Integer.valueOf(environment.getProperty("app.session.timeout.ms")) + 1 );
propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, "group1");
propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
return propsMap;
}
@Bean
KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
final ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(9);// was 3
factory.getContainerProperties().setPollTimeout(3000);
return factory;
}
And also in my listener I have: 而且在我的听众中,我有:
@KafkaListener(id = "baz", topics = "tipJobsForExecution", containerFactory="kafkaListenerContainerFactory")
public void listen(ConsumerRecord<?, ?> record)
My listener takes about 5 minutes to process the messages. 我的听众需要大约5分钟来处理消息。 As soon as it finished I read the following at kafka broker side:
完成后,我在kafka经纪人那边阅读了以下内容:
2018-05-03 10:29:11,210] INFO [GroupCoordinator 0]: Preparing to rebalance group baz with old generation 22 (__consumer_offsets-7) (kafka.coordinator.group.GroupCoordinator)
2018-05-03 10:29:11,210]信息[GroupCoordinator 0]:准备与22代老一代重新平衡组baz(__consumer_offsets-7)(kafka.coordinator.group.GroupCoordinator)
From my understanding kafka consideres the consumer dead and rebalances the group. 据我了解,kafka认为消费者死亡并重新平衡了群体。 My question is why is this happenning ?
我的问题是为什么会这样? One idea that I have is that maybe the heartbeat is not heartbeating every 3000ms as it should but I do not know how troubleshout this.
我的一个想法是,也许心跳并不是每隔3000ms就会进行一次心跳,但我不知道这会带来多大的麻烦。
Thanks in advance, Giannis 预先感谢,吉安尼斯
You have to be aware of three types of timeout configuration parameters for Kafka consumer. 您必须了解Kafka使用者的三种超时配置参数。
heartbeat.interval.ms - The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. heartbeat.interval.ms-使用Kafka的群组管理工具时,到达消费者协调员的心跳之间的预期时间。 Typically should be 1/3 of the session.timeout value Default value - 3000 ms
通常应为会话的1/3超时值默认值-3000毫秒
session.timeout.ms - If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a re-balance.Default value 10000 session.timeout.ms-如果代理在此会话超时之前未收到任何心跳信号,则代理将从该使用者中删除此使用者并启动重新平衡。默认值为10000
max.poll.interval.ms - If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will re-balance Default value - 300000 max.poll.interval.ms-如果在此超时到期前未调用poll(),则认为使用方失败,该组将重新平衡默认值-300000
In your case it looks like the poll interval is set to a too low value. 在您的情况下,轮询间隔似乎设置为太低的值。
Reference - https://kafka.apache.org/documentation/#newconsumerconfigs 参考-https: //kafka.apache.org/documentation/#newconsumerconfigs
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.