云中的卡夫卡消费者

Question

I am using spring kafka consumers which are wrapped in a spring boot app and deployed as microservices in pivotal cloud foundry. I am deploying 15 instances of the application & 15 kafka partitions. Since most of these instances starts up one by one , will there be multiple rebalancing performed? What is recommended way to deploying kafka consumers in cloud?

Answer 1

The new 0.11 client has a new property group.initial.rebalance.delay.ms see Notable changes in 0.11.0.0 .

A new config, group.initial.rebalance.delay.ms , was introduced. This config specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance. The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms . The default value for this is 3 seconds. During development and testing it might be desirable to set this to 0 inorder to not delay test execution time.

We are in the process of adding support for the 0.11 client. Currently it's planned for the 2.0 Spring for Apache Kafka release, but we are considering a 1.3 release with a subset of the 2.0 features, together with support for 0.11.

You could try using the new client with an older spring-kafka; it may work but, of course, you won't get 0.11 features such as transactions and Headers .