Kafka 代理從集群中移除且重新分配失敗后降低主題復制因子

Question

主題復制因子已增加到 45，而集群中可用的 Kafka 代理數量為 40。

這是由於重復的分區重新分配被停止而發生的。

kafka-topics --topic top --zookeeper zoo_url --describe

節目

Partition: 0 Leader: 20464 Replicas: 20464,20765,1882,20870,873,898,20752,16789,17181,20743,20854,20762,894,20459,20851,21070,20757,20766,20763,890,21173,20852,895,21314,20767,883,20467,16787,21071,20750,887,20760,7067,876,20764,891,20768,4880,20769,16788,20756,886,21172,1582,871,16827 Isr: 20464,20765,1882,20870,873,898,20752,16789,17181,20743,20762,894,20459,21070,20757,20766,20763,890,21173,895,21314,20767,883,20467,16787,20750,887,20760,7067,876,20764,891,20768,4880,20769,16788,20756,886,21172,871,16827...

一些副本不是集群的一部分。

跑步：

kafka-reassign-partitions --zookeeper zoo_url --topics-to-move-json-file assign.json --generate --broker-list...

失敗並出現以下錯誤

Partitions reassignment failed due to replication factor: 45 larger than available brokers: 21
kafka.admin.AdminOperationException: replication factor: 45 larger than available brokers: 21
    at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:117)
    at kafka.admin.ReassignPartitionsCommand$$anonfun$generateAssignment$1.apply(ReassignPartitionsCommand.scala:110)
    at kafka.admin.ReassignPartitionsCommand$$anonfun$generateAssignment$1.apply(ReassignPartitionsCommand.scala:108)
    at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
    at kafka.admin.ReassignPartitionsCommand$.generateAssignment(ReassignPartitionsCommand.scala:108)
    at kafka.admin.ReassignPartitionsCommand$.generateAssignment(ReassignPartitionsCommand.scala:91)
    at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:50)
    at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

--broker-list參數輸入是在線經紀人的 ID。

如何強制降低主題復制因子？

唯一有效的解決方案是通過運行降低一個分區的復制因子

kafka-reassign-partitions --zookeeper zoo_url --reassignment-json-file /tmp/assign.json --execute

當/tmp/assign.json如下所示

{ "partitions": [ { "partition": 0, "replicas": [20743,20762,894,20459,20757,895,20467,20760], "topic": "topic" } ], "version": 1 }

然后重新運行分區分配（生成分配並執行它）

Kafka 0.9.0.1作為 Cloudera 的一部分進行部署。

Answer 1

您可以在{ "partitions": [列表中添加多個分區，但使用kafka-reassign-partitions是處理此問題的唯一內置方法。 可能有外部工具可以設法以編程方式生成 JSON，但在嘗試查找當前分配中的未知代理 ID 時，它們也可能出錯