在 Camel-Kafka 中创建并发 Kafka 消费者

Question

I'm using camel-kafka version 2.14.3 . Below is the kafka URI:

<from uri="kafka:{brokerlist}?topic={topic-name}&amp;zookeeperHost={zookeeperHost}&amp;zookeeperPort={zookeeperPort}&amp;groupId={groupId-name}&amp;consumerStreams=2" />

note that I have used consumerStream=2 in URI options. But when I publish multiple messages to the topic topic-name at once (all to the same partition), the kafka consumer receives those messages sequentially. How can receives those messages parallely?

I'm looking for a solution like below:

<from uri="ibm_bean_name:queue_name?concurrentConsumers=2" /> 

is what I use to read concurrently from an ibm MQ

Answer 1

The partition is Kafka's unit of parallelism. Each partition will be assigned to a single consumer in a given consumer-group. You said you are producing to a single partition. That means all the messages will go to a single consumer, regardless of how many consumers you have in the group. If you want the messages to go to different consumers, you need to produce to different partitions.

Answer 2

If you have N partitions, then you can have up to N consumers within the same consumer group each of which reading from a single partition. When you have less consumers than partitions, then some of the consumers will read from more than one partition. Also, if you have more consumers than partitions then some of the consumers will be inactive and will receive no messages at all.

If you have one consumer per partition, then some of the partitions might receive more messages and this is why some of your consumers might be idle while some others might still processing some messages. Note that messages are not always inserted into topic partitions in a round-robin fashion as messages with the same key are placed into the same partition.