如何使Kafka消费者订阅新分区

Question

I have a couple of Kafka consumers subscribed to a topic and need to add some partitions. I'm playing around in the staging environment and add some new partitions. My understanding of Kafka is that they should automatically rebalance and they subscribe to the new partitions. Except they don't, restarting doesn't help. Each time I've done this (twice now) magically overnight the consumers subscribe to the new partitions.

If I create a new test group in a console it subscribes to the new partitions fine. But subscribing with the existing group id in the console, rebalances the workers but doesn't include the new partitions.

Why are they not subscribing when the partitions are created? And what is happening overnight that is causing them to start seeing the new partitions?

Answer 1

By default, clients (consumers and producers) refresh their metadata every 5 minutes. This is determine by metadata.max.age.ms .

In your example, consumers will find new partitions only when refreshing their metadata. So you either need to set this setting to a lower value or wait a bit.