Apache Kafka 分区未使用 RoundRobin 分区程序均匀分布

Question

I am using Kafka Producer with the RoundRobin partitioner to a topic which has 12 partitions.

Code can be found here https://github.com/apache/kafka/blob/2.8/clients/src/main/java/org/apache/kafka/clients/producer/RoundRobinPartitioner.java

The issue which i am facing is that this partitioner is giving the partition to send a particular message correctly(in a round robin way) but in the kafka producer code the partition method is getting called twice at line no 931 and 956(inside a if block for new batch) due to which certain partitions have no records sent to them and i cannot achieve the parallelism of 12 which i would like. I have tried the following thing. I have written a custom partitioner effectively with the same logic as roundrobin partitioner, only difference being if the partition method is called after newBatch method is invoked on the partitioner then the previously returned partition no is returned. I am kind of nervous of using this in production without understanding why was the kafka producer code the way it is as specified above and if someone can throw some light on it, i would really appreciate it. Also if anyone has any suggestions through which I can ensure that records get evenly distributed on every partition, i am open to them too.

The Kafka Producer Code - https://github.com/apache/kafka/blob/2.8/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java

Thank you in advance.

Answer 1

Solution

You should use DefaultPartitioner rather than RoundRobinPartitioner .

Comments on DefaultPartitioner.java says:

The default partitioning strategy:

If a partition is specified in the record, use it

If no partition is specified but a key is present choose a partition based on a hash of the key

If no partition or key is present choose the sticky partition that changes when the batch is full.

See KIP-480 for details about sticky partitioning.

Provide no specified partition number or partition key for your producer record, then the sticky partition will work, which approximatively make records on your Kafka partitions even. see Tip #2: Learn about the new sticky partitioner in the producer API

Internal

Again, I want to explain why RoundRobinPartitioner always not working in a generally supposed round robin way. "partition()" in RoundRobinPartitioner can only make sure that the number distribution of calling partition() method, is in a round robin way, which is not enough to ensure our records on partitions are even.

Pay attention to the fact that invocation of partition() in KafkaProducer.doSend() is strange(possible two consecutive invocation of partition()).

Subtle codes here may cause Unequal distribution of partitions when partitions number are even. Say we have 4 partitions(0,1,2,3), and 8 records.

record 1 -> 2 **partition()** call(return 0, return 1), finally assigned to partition 1
record 2 -> 2 **partition()** call(return 2, return 3), finally assigned to partition 3
record 3 -> 2 **partition()** call(return 0, return 1), finally assigned to partition 1
record 4 -> 2 **partition()** call(return 2, return 3), finally assigned to partition 3
record 5 -> 2 **partition()** call(return 0, return 1), finally assigned to partition 1
....

See? records will only be distributed to partition 1 and 3 !

Conclusion

RoundRobinPartitioner has one confusing name, it offers RoundRobin of calling partition() , rather than RoundRobin of KafkaProducer.send() . To make sure records equally distributed in all partitions, use DefaultPartitioner please!