What happens to existing topic's partitions when a new broker is added to the Kafka cluster?

Question

Let's say you have a one node Kakfa cluster and you create a topic with 2 partitions, because you want to be able to consume in parallel.

Then a new broker joins the cluster.

Will this trigger a redistribution of existing topic's partitions too, I mean, will all the data from the second partition of our topic be moved to the second broker ?

Or will the new broker be involved only in the future topic's partitions distribution ?

Answer 1

Kafka does not automatically redistribute existing partitions when brokers are added to aa cluster.

This is for a few reasons:

• Moving partitions can cause a lot of inter broker traffic and extra load on the cluster.

• It's hard to determine which partitions to move. In your case, it's trivial but imagine a cluster with 1000s of partitions and many brokers.

There is a section in the Kafka docs regarding cluster expansion .

There are a few tools to perform such an operation:

• kafka-reassign-partitions.sh is the default Kafka tool that can be used to rebalance partitions across new brokers in a controlled way.

• https://github.com/DataDog/kafka-kit

• https://github.com/linkedin/cruise-control