[英]Decrease topic replication factor after Kafka brokers removed from cluster and failed reassignments
The topic replication factor has increased to 45 while the number of available Kafka brokers in the cluster is 40.主题复制因子已增加到 45,而集群中可用的 Kafka 代理数量为 40。
This happened due to repeated stuck partition reassignments which were stopped.这是由于重复的分区重新分配被停止而发生的。
kafka-topics --topic top --zookeeper zoo_url --describe
shows节目
Partition: 0 Leader: 20464 Replicas: 20464,20765,1882,20870,873,898,20752,16789,17181,20743,20854,20762,894,20459,20851,21070,20757,20766,20763,890,21173,20852,895,21314,20767,883,20467,16787,21071,20750,887,20760,7067,876,20764,891,20768,4880,20769,16788,20756,886,21172,1582,871,16827 Isr: 20464,20765,1882,20870,873,898,20752,16789,17181,20743,20762,894,20459,21070,20757,20766,20763,890,21173,895,21314,20767,883,20467,16787,20750,887,20760,7067,876,20764,891,20768,4880,20769,16788,20756,886,21172,871,16827...
Some of the replicas are not part of the cluster.一些副本不是集群的一部分。
Running:跑步:
kafka-reassign-partitions --zookeeper zoo_url --topics-to-move-json-file assign.json --generate --broker-list...
fails with below error失败并出现以下错误
Partitions reassignment failed due to replication factor: 45 larger than available brokers: 21
kafka.admin.AdminOperationException: replication factor: 45 larger than available brokers: 21
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:117)
at kafka.admin.ReassignPartitionsCommand$$anonfun$generateAssignment$1.apply(ReassignPartitionsCommand.scala:110)
at kafka.admin.ReassignPartitionsCommand$$anonfun$generateAssignment$1.apply(ReassignPartitionsCommand.scala:108)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
at kafka.admin.ReassignPartitionsCommand$.generateAssignment(ReassignPartitionsCommand.scala:108)
at kafka.admin.ReassignPartitionsCommand$.generateAssignment(ReassignPartitionsCommand.scala:91)
at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:50)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
--broker-list
argument input are ids of online brokers. --broker-list
参数输入是在线经纪人的 ID。
How to force decreasing of topic replication factor?如何强制降低主题复制因子?
The only solution that worked is decreasing replication factor of one partition by running唯一有效的解决方案是通过运行降低一个分区的复制因子
kafka-reassign-partitions --zookeeper zoo_url --reassignment-json-file /tmp/assign.json --execute
when /tmp/assign.json
is like below当
/tmp/assign.json
如下所示
{ "partitions": [ { "partition": 0, "replicas": [20743,20762,894,20459,20757,895,20467,20760], "topic": "topic" } ], "version": 1 }
And then rerunning partition assignment (generation of assignment and executing it)然后重新运行分区分配(生成分配并执行它)
Kafka 0.9.0.1
is deployed as part of Cloudera. Kafka
0.9.0.1
作为 Cloudera 的一部分进行部署。
You can add more than one partition in the { "partitions": [
list, but using kafka-reassign-partitions
is the only built-in way to handle this.您可以在
{ "partitions": [
列表中添加多个分区,但使用kafka-reassign-partitions
是处理此问题的唯一内置方法。 There may be external tools that can manage to programmatically generate that JSON, but they may also error when trying to lookup unknown broker ids that are currently in the assignment可能有外部工具可以设法以编程方式生成 JSON,但在尝试查找当前分配中的未知代理 ID 时,它们也可能出错
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.