同一代理中同一主題的 Kafka 多個分區

Question

在深入研究 Kafka 時，我想到了如下問題。

Can Kafka have multiple partitions of the same topic in the same broker?

起初，我認為我會不受歡迎，因為它與分區的目的相矛盾——為了高可用性而進行冗余。 但在某種程度上，沒有理由不提供這個。

這在卡夫卡可能嗎？ 如果是這樣，這種設置在實踐中是否經常使用？

Answer 1

Kafka 可以在同一個代理中擁有同一個主題的多個分區嗎？

是的，完全可以。 例如：

╰─$ ./bin/kafka-topics.sh --bootstrap-server localhost:9092 \
                          --create --topic rmoff-test \
                          --partitions 6 \
                          --replication-factor 1
Created topic rmoff-test.

╭─rmoff@asgard03 ~/Downloads/kafka_2.13-3.1.0
╰─$ kcat -b localhost:9092 -L
Metadata for all topics (from broker 1: localhost:9092/1):
 1 brokers:
  broker 1 at localhost:9092 (controller)
 1 topics:
  topic "rmoff-test" with 6 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1
    partition 1, leader 1, replicas: 1, isrs: 1
    partition 2, leader 1, replicas: 1, isrs: 1
    partition 3, leader 1, replicas: 1, isrs: 1
    partition 4, leader 1, replicas: 1, isrs: 1
    partition 5, leader 1, replicas: 1, isrs: 1

您可能混淆了分區（用於嚴格排序和高吞吐量）和分區副本（用於可用性/容錯）的概念。

一個代理上不能有多個副本。 如果您嘗試創建一個主題，其副本數多於可用代理數，它將失敗：

╰─$ # There is one broker running 
╰─$ kcat -b localhost:9092 -L | grep broker
Metadata for all topics (from broker 1: localhost:9092/1):
 1 brokers:
  broker 1 at localhost:9092 (controller)

╰─$ # Now try to create a topic with _one_ partition but _two_ replicas:
╰─$ ./bin/kafka-topics.sh --bootstrap-server localhost:9092 \
                          --create --topic rmoff-test-2 \
                          --partitions 1 \
                          --replication-factor 2                                                          

Error while executing topic command : Unable to replicate the partition 2 time(s): The target replication factor of 2 cannot be reached because only 1 broker(s) are registered.
[2022-03-11 09:51:33,721] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Unable to replicate the partition 2 time(s): The target replication factor of 2 cannot be reached because only 1 broker(s) are registered.
 (kafka.admin.TopicCommand$)