Sharing messages between kafkacat consumers

Question

This is pretty much the same question as this: Messages sent to all consumers with the same consumer group name . The accepted answer is to use Kafka 0.8.1 or newer, which I did.

Kafka documentation says :

If all the consumer instances have the same consumer group, then this works just like a traditional queue balancing load over the consumers.

But I am not able to observe this behaviour using Kafka 0.8.2.1 and kafkacat .

My setup:

• Kafka + Zookeeper running in spotify/kafka container (via boot2docker)
• One producer
• Two consumers with the same group.id

First, in running Kafka container, I created a topic:

$KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic beta

Two consumers subscribed to same topic:

kafkacat -C -b $(boot2docker ip):9092 -t beta -X group.id=mygroup

Then I produce a message using kafkacat:

date | kafkacat -P -b $(boot2docker ip):9092 -t beta

I expect that only one consumer receives the message, but actually both of them do. What am I doing wrong?

Edit: When I try to run the same consumers using kafka-console-consumer.sh , all is fine:

echo "group.id=mygroupid" > /consumer.beta.properties

$KAFKA_HOME/bin/kafka-console-consumer.sh \
  --zookeeper localhost:2181 \
  --topic beta \
  --consumer.config /consumer.beta.properties

All works as expected: the message is only consumed once. I assume the issue lies with kafkacat.

Answer 1

kafkacat现在支持0.9高级KafkaConsumer：

kafkacat -b <broker> -G <groupid> topic1 topic2..

Answer 2

It turns out this feature is not supported by librdkafka (the library that kafkacat is using). Quoting from the Github issue :

Short answer, this is not supported for now. Also, whatever you see in the Kafka documentation is only true if you're using the JVM consumer/producer. Assume that any other consumer/producer implementation is not entirely following the doc.