[英]Kafka | Increase replication factor of multiple topics
I have a 3 broker Kafka cluster with many topics with replication factor 1. I know I can increase it by passing JSON file with the partition reassignment configuration to kafka-reassign-partitions.sh
. 我有一个3代理Kafka集群,其中的主题很多,复制因子为1。我知道可以通过将带有分区重新分配配置的JSON文件传递给kafka-reassign-partitions.sh
来增加它。
My confusion is should I pass a single JSON file with partition reassignment details of all topics or should I create a JSON for each topic and run them individually? 我的困惑是我应该传递一个带有所有主题的分区重新分配详细信息的JSON文件,还是应该为每个主题创建一个JSON并分别运行它们?
You can either create multiple .json
files or use a single file that contains reassignment details for more than one topic: 您可以创建多个.json
文件,也可以使用一个文件包含多个主题的重新分配详细信息:
{
"version":1,
"partitions":[
{"topic":"topic_1","partition":0,"replicas":[0,1]},
{"topic":"topic_1","partition":1,"replicas":[1,0]},
{"topic":"topic_2","partition":0,"replicas":[0,1]},
{"topic":"topic_2","partition":1,"replicas":[1,0]}
]
}
And then run 然后跑
./bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
Your topics should look like below: 您的主题应如下所示:
./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic_1
Topic:demo-topic PartitionCount:2 ReplicationFactor:2 Configs:
Topic: topic_1 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: topic_1 Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1,0
./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic_2
Topic:demo-topic PartitionCount:2 ReplicationFactor:2 Configs:
Topic: topic_2 Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: topic_2 Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1,0
Finally, Finally, the --verify option can be used with the tool to check the status of the partition reassignment. 最后,最后,--verify选项可与该工具一起使用,以检查分区重新分配的状态。 Note that the same expand-cluster-reassignment.json (used with the --execute option) should be used with the --verify option 请注意,应将相同的expand-cluster-reassignment.json(与--execute选项一起使用)与--verify选项一起使用
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [topic_1,0] completed successfully
Reassignment of partition [topic_1,1] is in progress
Reassignment of partition [topic_2,0] completed successfully
Reassignment of partition [topic_2,1] completed successfully
This a balance of cost / risk. 这是成本/风险的平衡。
Reassigning all topics together: 重新分配所有主题:
Reassigning topics in "small" chunks: 以“较小”的块重新分配主题:
Depending on the size and usage of your cluster, you should be able to identify which method is the best for you. 根据群集的大小和使用情况,您应该能够确定哪种方法最适合您。 In a busy cluster, I'd recommend setting reasignment quotas and only reassigning topics by chunks as otherwise reassignment will try to execute as fast as possible and this can impact the cluster greatly. 在繁忙的群集中,我建议设置重新分配配额,并且仅按块重新分配主题,否则重新分配将尝试尽快执行,这会极大地影响群集。 If your cluster is mostly fresh/unused then you may be able to reassign all topics at the same time. 如果您的群集大部分都是新的/未使用的,则可以同时重新分配所有主题。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.