Kafka old consumer rebalance issue

Question

In our system we're using an older version of kafka (0.9.0.1) and the old scala consumer API in a tomcat application. Everything works fine most of the time, but sometimes when the servers where the consumers run are heavily utilised by some other tasks in the app then the consumers become unresponsive which triggers as expected a rebalance and that consumer is removed from its partitions and other consumers are used.

My question is if there is an easy way for the consumer to re-register itself when it comes back up?

I know that the old consumers store the partition consumer details in Zookeeper and was thinking we could have a task that would periodically check if our consumer is registered there and restart the consumer if not, but I'm not sure what exactly we should check there. Can anyone point me to some documentation about the data stored in zookeeper by kafka (haven't found anything in the official documentation sadly :( )?

Answer 1

Basically, what you want is fixed assignments, and that consumer groups never rebalance. If there was a way to disable automatic consumer rebalancing in the old Scala client, or maybe even increase the rebalance timeout to a much higher value, that could also work, but I couldn't find how to do that with the old Scala consumer.

However, it is possible to assign fixed topic/partitions when using the newer Java consumers, also available in that same 0.9 kafka version. Look for Subscribing To Specific Partitions in the latest Javadocs:

https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html

Subscribing To Specific Partitions
   In the previous examples we subscribed to the topics we were interested in and 
   let Kafka give our particular process a fair share of the partitions for those topics. 
   This provides a simple load balancing mechanism so multiple instances of our program 
   can divided up the work of processing records.

   In this mode the consumer will just get the partitions it subscribes to 
   and if the consumer instance fails no attempt will be made to 
   rebalance partitions to other instances.