Selective Kafka rebalancing on Kubernetes infrastructure

Question

I am running a kafka cluster with a set of consumers on a dockerized Kubernetes infrastructure. The typical workflow is that when a certain consumer (of the consumer group) dies, a rebalancing process will be triggered, and a new assignment of the partitions to the set of the consumers (excluding the failed one) is performed.

After some time, Kubernetes controller will recreate/restart the consumer instance that has failed/died and a new rebalance is performed again.

Is there any way to control the first rebalancing process (when the consumer died) eg, such as to wait few seconds without rebalancing until the failed consumer returns, or until a time out is triggered. And if the consumer returned, continue consuming based on the old rebalancing assignment (ie, without new rebalancing)?

Answer 1

There are the 3 parameter on that basis group coordinator decide consumer is dead or alive

session.timeout.ms
max.poll.interval.ms
heartbeat.interval.ms

You can avoid unwanted rebalancing by tuning above three parameter and one thumb rule: used separate thread for calling 3rd party api in pool loop.

tuning above three parameter required ans. of below questions

1. what is size max.poll.records
2. How much avg.time application is taking to process 1 record[message]
3. How much avg.time application is taking to process complete batch

Please refer Kafka consumer config

https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html

you can also explore Cooperative Rebalances

https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/