Kafka streams: Multiple stream applications for multiple topic partitions

Question

I have an input topic with 2 partitions and 2 kafka stream applications with same application.id and different sink topics.

1) I start both applications together and they start receiving data but when I kill one of the applications, the other one keeps getting stuck at this. (FYI,The consumer setting has SESSION_TIMEOUT_MS_CONFIG set to 120000)

[2017-10-08 00:26:17,297] DEBUG [kafka-coordinator-heartbeat-thread | ]Sending Heartbeat request for group test.joins.hourly to coordinator :9092 (id: 2147483647 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [2017-10-08 00:26:17,398] DEBUG [kafka-coordinator-heartbeat-thread | test.joins.hourly] Received successful Heartbeat response for group test.joins.hourly (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

2) How can I make sure that the second application starts processing records from both partitions? How can I prove this?

Please help!

Answer 1

What do you mean by "the other one is stuck"? How long have you waited for it to start consuming?

In your case it might take 2 minutes for cluster to rebalance your consumers after you kill one of them. It also depends on heartbeat.interval.ms - the smaller it is, the smaller average time to rebalance will be.