Kafka multiple partitions of the same topic in the same broker

Question

While diving deep into Kafka, a question like below came to mind.

Can Kafka have multiple partitions of the same topic in the same broker?

At first, I thought I would be undesirable since it contradicts to the purpose of partitions - making redundancy for high availability. But in some way, there are no reasons not to supply this.

Is this possible in Kafka? If so, is this kind of setting used a lot in practice?

Answer 1

Can Kafka have multiple partitions of the same topic in the same broker?

Yes, it totally can. For example:

╰─$ ./bin/kafka-topics.sh --bootstrap-server localhost:9092 \
                          --create --topic rmoff-test \
                          --partitions 6 \
                          --replication-factor 1
Created topic rmoff-test.

╭─rmoff@asgard03 ~/Downloads/kafka_2.13-3.1.0
╰─$ kcat -b localhost:9092 -L
Metadata for all topics (from broker 1: localhost:9092/1):
 1 brokers:
  broker 1 at localhost:9092 (controller)
 1 topics:
  topic "rmoff-test" with 6 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1
    partition 1, leader 1, replicas: 1, isrs: 1
    partition 2, leader 1, replicas: 1, isrs: 1
    partition 3, leader 1, replicas: 1, isrs: 1
    partition 4, leader 1, replicas: 1, isrs: 1
    partition 5, leader 1, replicas: 1, isrs: 1

It's possible you're mixing up the concept of partitions (used for strict ordering and high throughput) with that of partition replicas (used for availability/fault tolerance).

You cannot have multiple replicas on one broker. If you try to create a topic with more replicas than available brokers it will fail:

╰─$ # There is one broker running 
╰─$ kcat -b localhost:9092 -L | grep broker
Metadata for all topics (from broker 1: localhost:9092/1):
 1 brokers:
  broker 1 at localhost:9092 (controller)

╰─$ # Now try to create a topic with _one_ partition but _two_ replicas:
╰─$ ./bin/kafka-topics.sh --bootstrap-server localhost:9092 \
                          --create --topic rmoff-test-2 \
                          --partitions 1 \
                          --replication-factor 2                                                          

Error while executing topic command : Unable to replicate the partition 2 time(s): The target replication factor of 2 cannot be reached because only 1 broker(s) are registered.
[2022-03-11 09:51:33,721] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Unable to replicate the partition 2 time(s): The target replication factor of 2 cannot be reached because only 1 broker(s) are registered.
 (kafka.admin.TopicCommand$)