卡夫卡主题与分区

Question

Simple question:

Let's assume I have a topic with 3 partitions: Topic: StateEvents P1, P2 and P3.

Let's also assume that the producer generates 20 messages:

1, 2, 3, ..........20

my question is:

When the producer produces these messages:

1) Will each message be in only and only 1 partition? that is, 1 in P1, 2 in P2, 3 in P3, then 4 in P1, 5 in P2, 6 in P3 and so on and so forth?

2) If # 1 is true, when a consumer subscribes, it would be subscribing to ALL partitions so that it gets all messages?

Thanks

Answer 1

1. Yes, each message is written in only one partition.

2. When single consumer subscribes to the kafka topic(having multiple partitions), it will read messages from all partitions. But if you run multiple consumers with the same consumer group.id, each consumer will read from different-different partitions.

Let's say, a kafka topic has 3 partitions, and you have 3 consumers with same group.id. Each consumer will read from 1-1 partitions. But if there is only one consumer, it will read from all 3 partitions.

Answer 2

1) The destination partition is determined by the producer. With the default partitioner algorithm (which can be customized) the destination partition is : hash(message-key) % num_partitions. It means that all messages with same key goes into the same partition. So if you are using a key and all messages have the same key, then they are going to the same partition. If key is not specified a round robin is used. In any case a message goes always to only ONE partition.

2) if the consumer is the only one in the consumer group, it will get all the partitions. You can add more consumers in the same consumer group for sharing the load (in your case up to 3 consumers, which is the number of partitions you have in the topic, so that one consumer gets one partition).

Having different consumers getting messages from different partitions is the way how Kafka scale really well. It's not a drawback because you have to think in terms of consuming application (made by multiple consumers). The application identifier can be the group-id used by all its consumers: you have the application getting ALL the messages from the topic but the load is spread across its consumers.

Answer 3

Each message will be sent to only one partition

If key is not null, partition ID calculation is implemented according to the partition method implemented in the configuration. here is the source code

class DefaultPartitioner(props: VerifiableProperties = null) extends Partitioner {
  private val random = new java.util.Random

  def partition(key: Any, numPartitions: Int): Int = {
    Utils.abs(key.hashCode) % numPartitions
  }
}

To ensure the same type of message sequentiality (FIFO), a partition can only be consumed by one consumer of the same group, and consumers of different groups can bind the same partition for repeated consumption. But a consumer can cosume more than one partition.