[英]Python - Kafka: consumer failing
I have a simple Producer-Consumer setup: 1 producer(as a thread) and 2 consumers(as 2 processes). 我有一个简单的Producer-Consumer设置:1个生产者(作为一个线程)和2个消费者(作为2个进程)。 The run method of producer:
生产者的运行方法:
def run(self):
producer = KafkaProducer(bootstrap_servers=self.bootstrap_servers,
api_version=(0, 10))
while not self.stop_event.is_set():
self.logger.info("Checking for new changes")
self.check_for_new_changes(producer)
self.logger.info("Sleeping for {minutes}
minutes...".format(minutes=self.time_to_sleep / 60))
time.sleep(self.time_to_sleep)
producer.close()
Basically it checks for changes, sends messages if new changes found and then goes to sleep for 5 minutes. 基本上,它检查更改,如果发现新更改则发送消息,然后进入睡眠状态5分钟。
The run method: 运行方法:
def run(self):
if self.group_id:
consumer = KafkaConsumer(bootstrap_servers=self.bootstrap_servers,
consumer_timeout_ms=1000,
api_version=(0, 10),
group_id=self.group_id)
else:
consumer = KafkaConsumer(bootstrap_servers=self.bootstrap_servers,
consumer_timeout_ms=1000,
api_version=(0, 10))
consumer.subscribe(['new_change'])
while not self.stop_event.is_set():
for msg in consumer:
self.logger.info("New message:\n{msg}".format(msg=msg))
self.process_new_change(json.loads(msg.value))
if self.stop_event.is_set():
consumer.close()
return
consumer.close()
It seems to work fine but after running for a while I get these messages in the coordinator log: 似乎工作正常,但运行了一段时间后,我在协调器日志中收到了以下消息:
[2017-12-17 02:06:40,639] INFO [GroupCoordinator 0]: Member kafka-python-1.3.5-f5cdcad3-bc1a-4623-a42b-f5de5e8bded1 in group meta_data_consumer has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2017-12-17 02:06:40,659] INFO [GroupCoordinator 0]: Preparing to rebalance group meta_data_consumer with old generation 15 (__consumer_offsets-6) (kafka.coordinator.group.GroupCoordinator)
[2017-12-17 02:06:40,659] INFO [GroupCoordinator 0]: Group meta_data_consumer with generation 16 is now empty (__consumer_offsets-6) (kafka.coordinator.group.GroupCoordinator)
[2017-12-17 02:06:41,784] INFO [GroupCoordinator 0]: Member kafka-python-1.3.5-bdea8ce3-922f-4ee1-9959-13341e1730f5 in group failures_consumer has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2017-12-17 02:06:41,785] INFO [GroupCoordinator 0]: Preparing to rebalance group failures_consumer with old generation 9 (__consumer_offsets-35) (kafka.coordinator.group.GroupCoordinator)
[2017-12-17 02:06:41,785] INFO [GroupCoordinator 0]: Group failures_consumer with generation 10 is now empty (__consumer_offsets-35) (kafka.coordinator.group.GroupCoordinator)
This kills my consumers and they stop running. 这杀死了我的消费者,他们停止了运营。 I don't see any exceptions or errors in the consumer logs.
我在使用者记录中看不到任何例外或错误。
What might cause them to fail? 是什么导致他们失败?
I think your zookeeper.session.timeout.ms
is set lower than 5 minutes. 我认为您的
zookeeper.session.timeout.ms
设置为低于5分钟。 Adjust that timeout from zookeeper settings. 从Zookeeper设置中调整超时时间。 See if it still fails.
查看它是否仍然失败。 If it does, then you should adjust the timeouts in kafka configs.
如果是这样,那么您应该在kafka配置中调整超时。
group.max.session.timeout.ms
, rebalance.timeout.ms
, heartbeat.interval.ms
should be adjusted accordingly. group.max.session.timeout.ms
, rebalance.timeout.ms
, heartbeat.interval.ms
应作相应调整。 Your client sleeps for 5 minutes, and during that time, one of those timeout value is exceeded, and Group Coordinator tries to rebalance the consumer, thinking those consumers have failed. 您的客户睡眠了5分钟,在这段时间内,超过了其中一个超时值,并且组协调器尝试重新平衡消费者,认为这些消费者失败了。
Source: Kafka Documentation 资料来源: Kafka文档
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.