我可以在 Kafka 上自动重新分配分区吗？

Question

I 'm setting up a Kafka cluster with 3 nodes on Kubernetes. If a node crashes, should I auto-reassgin partitions by scripts? Thank you very much.

Answer 1

As @asolanki mentioned, Kafka will automatically failover leaders to brokers that are alive. I just wanted to note that this is not called a reassignment - it's called a leader failover .

In Kafka, a partition is assigned to N number of replicas (N=replication factor) . The in-sync replica set (ISR) is the set of replicas that are in-sync (alive and have the latest data). The ISR naturally follows the replica set.

A reassignment is when you change the replicas that this partition should be hosted on. I think an example would make this clear:

A simple failover :

1. partition=0 is on brokers (replicas) replicas=[1,2,3], isr=[1,2,3]. Broker 1 is the leader
2. broker 1 fails. Kafka automatically fails over to another broker
3. replicas=[1,2,3], isr=[2,3] (broker 1 is dead)

A simple reassignment :

1. partition=0 is on brokers (replicas) replicas=[1,2,3], isr=[1,2,3]
2. We reassign partition=0 to replicas=[4,5,6]
3. replicas=[4,5,6], isr=[4,5,6]

Answer 2

Kafka takes care of managing the partitions if replication for a partition is set up correctly. Every topic partition in Kafka is replicated n times, where n is the replication factor of the topic. So in short if you have set up replication no action is required, pod can go down and back up, the producers and consumers are not affected.

For a longer description of how this works look at this