Kafka不会在主题中均匀填充分区

Question

I want to have 1 topic with 10 partitions. I am using default configuration of Kafka. I create 1 topic with 10 paritions by that helper script and now I am about to produce messages to it.

The thing is that it seems like there are only 5 partitions from which consumers fetches data.

Let's describe it in more detail.

I know that common stuff that you need one consumer thread per partition. I want to be able to commit offsets per partition and this is possible only when I have 1 thread per consumer connector per partition (I am using high level consumer).

So I create 10 threads, in each thread I am calling Consumer.createJavaConsumerConnector() where I am doing this

topicCountMap.put("mytopic", 1);

and in the end I have 1 iterator which consumes messages from 1 partition.

When I do this 10 times, I have 10 consumers, consumer per thread per partition where I can commit offsets independently per partition because if I put different number from 1 in topic map, I would end up with more then 1 consumer thread for that topic so if I am about to commit offsets with created consumer instance, it would commit them for all threads which is not desired hence for multiple partitions which is not desired.

But the thing is that when I use consumers, only 5 consumers are involved and it seems that other threads are idle but I do not know why.

The first possible reason is that even I have 10 partitions, only 5 partitions have messages so other 5 consumers are idle, but I do not understand how it is possible that messages are not spread evenly accross all partitions when I am using producers. I am sending like 1M of messages so if it is said they are spread evenly, every paritition has to have at least some message in it.

// EDIT

I managed to create 10 partitions in a topic but I have only 7 consumers. That's just a miracle to me.

The thing is that I am creating these consumer threads in a loop. So I start first thread (submit to executor service), then another, then another and so on.

So the scenario is that first consumer gets all 10 partitions, then 2nd connects so it is splits between these two to 5 and 5 (or something similar), then other threads are connecting.

I understand this as a partition rebalancing among all consumers so it behave well in such sense that if more consumers are being created, partition balancing occurs between these consumers so every consumer should have some partitions to operate upon.

But from the results I see that there is only 7 consumers and according to consumed messages it seems they are split like 3,2,1,1,1,1,1 partition-wise. Yes, these 7 consumers covered all 10 partitions, but why consumers with more then 1 partition do no split and give partitions to remaining 3 consumers?

I am pretty much wondering what is happening with remaining 3 threads and why they do not "grab" partitions from consumers which have more then 1 partition assigned.

Answer 1

I've seen similar behavior when I (accidentally) access the topic programmatically before creating the topic via admin script. In that situation, the number of partitions, as well as other topic configuration settings, defaults to the values in broker.config