简体繁体 English

Kafka流：用于多个主题分区的多个流应用程序

[英]Kafka streams: Multiple stream applications for multiple topic partitions

原文 2017-10-08 07:32:04 4 1 apache-kafka/ apache-kafka-streams

I have an input topic with 2 partitions and 2 kafka stream applications with same application.id and different sink topics. 我有一个输入主题，其中包含2个分区和2个具有相同application.id和不同接收器主题的kafka流应用程序。

1) I start both applications together and they start receiving data but when I kill one of the applications, the other one keeps getting stuck at this. 1）我同时启动了两个应用程序，它们开始接收数据，但是当我杀死其中一个应用程序时，另一个应用程序就陷入了困境。 (FYI,The consumer setting has SESSION_TIMEOUT_MS_CONFIG set to 120000) （仅供参考，使用者设置将SESSION_TIMEOUT_MS_CONFIG设置为120000）

[2017-10-08 00:26:17,297] DEBUG [kafka-coordinator-heartbeat-thread | [2017-10-08 00：26：17,297]调试[kafka-coordinator-heartbeat-thread | ]Sending Heartbeat request for group test.joins.hourly to coordinator :9092 (id: 2147483647 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [2017-10-08 00:26:17,398] DEBUG [kafka-coordinator-heartbeat-thread | 每小时将心跳请求发送给协调员：9092（标识号：2147483647机架：空）（org.apache.kafka.clients.consumer.internals.AbstractCoordinator）[2017-10-08 00：26：17,398]调试[kafka-coordinator-heartbeat-thread | test.joins.hourly] Received successful Heartbeat response for group test.joins.hourly (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) test.joins.hourly]收到针对组test.joins.hourly的成功心跳响应（org.apache.kafka.clients.consumer.internals.AbstractCoordinator）

2) How can I make sure that the second application starts processing records from both partitions? 2）如何确保第二个应用程序开始处理两个分区中的记录？ How can I prove this? 我如何证明这一点？

Please help! 请帮忙！

1 个解决方案

What do you mean by "the other one is stuck"? “另一个卡住了”是什么意思？ How long have you waited for it to start consuming? 您等了多久才开始消费？

In your case it might take 2 minutes for cluster to rebalance your consumers after you kill one of them. 在您的情况下，杀死一个用户后，群集可能需要2分钟才能重新平衡您的用户。 It also depends on heartbeat.interval.ms - the smaller it is, the smaller average time to rebalance will be. 它还取决于heartbeat.interval.ms越小，重新平衡的平均时间越短。