更改Kafka配置max.poll.interval.ms导致一名消费者卡在（重新）加入组中

Question

I am using Kafka version 0.10.2.0 and I have two VMs on cloud each has one Kafka consumer. These two consumers belong to the same consumer group. It works fine until I updated the Kafka consumer configuration max.poll.interval.ms to Integer.MAX_VALUE . When I restart the VMs, the VM starts first works fine. The VM started later will always stuck at (Re-)joining group

Does any one have idea what causes this?

I make this change because I saw in version 0.10.2.1, the default vaule of max.poll.interval.ms is changed from 300000 to Integer.MAX_VALUE .

Answer 1

It doesn't seem that the problem could be induced by the 'max.poll.interval.ms'

Based on the Kafka documentation ( https://kafka.apache.org/documentation/ )

The way consumption is implemented in Kafka is by dividing up the partitions in 
the log over the consumer instances so that each instance is the exclusive 
consumer of a "fair share" of partitions at any point in time. This process of 
maintaining membership in the group is handled by the Kafka protocol 
dynamically. If new instances join the group they will take over some 
partitions from other members of the group; if an instance dies, its partitions 
will be distributed to the remaining instances.

I'd check if there are sufficient partitions to distribute into those consumers.

Additionally, I'd recommend you to activate DEBUG mode in the Kafka logs. Normally, it is showing a lot of info about the internals of the joining process.

Answer 2

This was a change for the version 0.10.1.0:

The new Java Consumer now supports heartbeating from a background thread. There is a new configuration max.poll.interval.ms which controls the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default). The value of the configuration request.timeout.ms must always be larger than max.poll.interval.ms because this is the maximum time that a JoinGroup request can block on the server while the consumer is rebalancing, so we have changed its default value to just above 5 minutes.

which applies to your version also

BTW in version 0.10.2.0 they didn't change max.poll.interval.ms for kafka consumer but for kafka streams consumer:

The internal Kafka Streams consumer max.poll.interval.ms default value was changed from 300000 to Integer.MAX_VALUE