[英]Optimal number of partition for kafka topic on 5 brokers with replication factor=3 in 1 cluster
[英]Decrease topic replication factor after Kafka brokers removed from cluster and failed reassignments
主题复制因子已增加到 45,而集群中可用的 Kafka 代理数量为 40。
这是由于重复的分区重新分配被停止而发生的。
kafka-topics --topic top --zookeeper zoo_url --describe
节目
Partition: 0 Leader: 20464 Replicas: 20464,20765,1882,20870,873,898,20752,16789,17181,20743,20854,20762,894,20459,20851,21070,20757,20766,20763,890,21173,20852,895,21314,20767,883,20467,16787,21071,20750,887,20760,7067,876,20764,891,20768,4880,20769,16788,20756,886,21172,1582,871,16827 Isr: 20464,20765,1882,20870,873,898,20752,16789,17181,20743,20762,894,20459,21070,20757,20766,20763,890,21173,895,21314,20767,883,20467,16787,20750,887,20760,7067,876,20764,891,20768,4880,20769,16788,20756,886,21172,871,16827...
一些副本不是集群的一部分。
跑步:
kafka-reassign-partitions --zookeeper zoo_url --topics-to-move-json-file assign.json --generate --broker-list...
失败并出现以下错误
Partitions reassignment failed due to replication factor: 45 larger than available brokers: 21
kafka.admin.AdminOperationException: replication factor: 45 larger than available brokers: 21
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:117)
at kafka.admin.ReassignPartitionsCommand$$anonfun$generateAssignment$1.apply(ReassignPartitionsCommand.scala:110)
at kafka.admin.ReassignPartitionsCommand$$anonfun$generateAssignment$1.apply(ReassignPartitionsCommand.scala:108)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
at kafka.admin.ReassignPartitionsCommand$.generateAssignment(ReassignPartitionsCommand.scala:108)
at kafka.admin.ReassignPartitionsCommand$.generateAssignment(ReassignPartitionsCommand.scala:91)
at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:50)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
--broker-list
参数输入是在线经纪人的 ID。
如何强制降低主题复制因子?
唯一有效的解决方案是通过运行降低一个分区的复制因子
kafka-reassign-partitions --zookeeper zoo_url --reassignment-json-file /tmp/assign.json --execute
当/tmp/assign.json
如下所示
{ "partitions": [ { "partition": 0, "replicas": [20743,20762,894,20459,20757,895,20467,20760], "topic": "topic" } ], "version": 1 }
然后重新运行分区分配(生成分配并执行它)
Kafka 0.9.0.1
作为 Cloudera 的一部分进行部署。
您可以在{ "partitions": [
列表中添加多个分区,但使用kafka-reassign-partitions
是处理此问题的唯一内置方法。 可能有外部工具可以设法以编程方式生成 JSON,但在尝试查找当前分配中的未知代理 ID 时,它们也可能出错
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.