Kafka is not sending messages to other partitions

Question

Apache Kafka installed on Mac (Intel). Single local producer and single local consumer. 1 topic with 3 partitions and 1 replication factor is created:

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic animal --partitions 3 --replication-factor 1

Producer code:

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic animal

Producer Messages:

>alligator
>crocodile
>tiger

When producing messages (manually via producer-console), all go into the same partition. Shouldn't they get distributed across partitions?

I've tried with 3 records (as above), but they get sent to 1 partition only. Checked within tmp/kafka-logs/topic-0/00** 00.log Other logs in topic- are empty.

I've tried with tens of records, but no luck.

I even increased the default partition configuration (num.partitions=3) within 'config/server.properties', but no luck.

I've also tried with different topics, but no luck.

Answer 1

Starting with kafka 2.4, the default partitioner was changed from round-robin to sticky , which will stick to the same partition (pun intended) for an entire batch.

With my kafka version, the kafka-console-producer uses a default batch size of 16384, so once you produce enough messages to fill that buffer, the partition will change.

Answer 2

If a producer, produces messages with the same key then it's guaranteed to be produced on the same partition. so in your case if you want it to be consumed by different partitions than make sure to publish it with different keys. You will need to set below property.

--property parse.key=true

See below command to produce record with key.

kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic --property parse.key=true --property key.separator=,

> key1,value1
> key2,value2