Kafka Brokers don't balance partitions after a broker goes down

Question

We have a 3-broker 3-zookeeper cluster and we've taken down a broker. We have total of 180 partitions, each of the topics have 2 replicas. When a node is taken down, there are 75 under replicated partitions and it stays that way and it doesn't look like anything happens. When I start up the broker I took down, the partitions are quickly picked up by it and it works ok.

The machines are quite big (30gb ram, fast disks) and all the data is 10gb on each broker so I have no idea why it wouldn't move the partitions quickly from a node that is down to a node that is still up, seems like it's not aware that the node was taken down.

Any tips? How can I monitor the recovery process after a node is taken down?

Kafka version - 2.6.0

Answer 1

This is by design, no data is moved off the broker until you manually move partitions off of it first using kafka-reassign-partitions

Similarly, you'd need to do this if you're trying to fully remove a node from the cluster, which is effectively the same behavior of having it crash and never come back