卡夫卡简单的消费者间歇性地丢失了消息

Question

I have a Kafka application from where I have been consuming messages using kafka-console-consumer.sh as following:

$./kafka-console-consumer.sh --zookeeper zookeeperhost:2181 --topic myTopic

which gives all the messages which I write to Kafka broker through a Kafka consumer without any miss.

Recently I deployed the application in a different environment where zookeeperhost is not accessible (due to some reason). So I am using kafka-simple-consumer-shell.sh instead as below:

$./kafka-simple-consumer-shell.sh --broker-list brokerhost:9092 --topic myTopic --partition 0 --max-messages 1

But with this I see few messages (around 2-4 in 5000) go missed. Could someone please explain how kafka-simple-consumer-shell.sh reads messages.

I am doubting that probably some messages are going to some different partition and as I am just reading from partition 0 so I am not getting all the messages every time. But I do not know how to check how many partitions are there? and what are the ids for other partitions? I tried with 1 but it does not work.

Could someone please help.

Answer 1

kafka-simple-consumer.sh simply creates a consumer that reads messages from one partition. So your command simply reads a single message in partition 0 of myTopic from brokerhost:9092 . If partition 1 is not in the same broker, it will not work as what you did. (For more information, check Code from GitHub )

If you can access to the Zookeeper host, you can simply check how partitions are distributed in a cluster with

bin/kafka-topics.sh --describe --zookeeper zookeeperhost:2181 --topic myTopic

but if you can't access to the Zookeeper host, there are two ways as I can think of.

1. Provide a list having all brokers as a parameter and try partition numbers from 0 to N. You can provide multiple brokers to --broker-list in a format of broker1:port2,broker2:port2,broker3:port3 . Then you can figure out how many partitions exist in the entire cluster, but still you don't know which broker has which partitions.
2. Manually check a log directory of each broker. Check /tmp/kafka-logs (if you are using a default log directory). You will find directories like myTopic-0 , myTopic-1 , ... which are in a format of topic-partition# . You can check which broker has which partitions manually with this.