如何从 Kafka 主题中获取最近的消息

Question

Do we have any option like fetching recent 10/20/ etc., messages from Kafka topic. I can see --from-beginning option to fetch all messages from the topic but if I want to fetch only few messages first, last, middle or latest 10. do we have some options?

Answer 1

First N messages

You can use --max-messages N in order to fetch the first N messages of a topic.

For example, to get the first 10 messages, run

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning  --max-messages 10

Next N messages

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --max-messages 10

Last N messages

To get the last N messages, you need to define a specific partition and the offset:

bin/kafka-simple-consumer-shell.sh --bootstrap-server localhost:9092 --topic test--partition testPartition --offset yourOffset

M to N messages

Again, for this case you'd have to define both the partition and the offset. For example, you can run the following in order to get N messages starting from an offset of your choice:

bin/kafka-simple-consumer-shell.sh --bootstrap-server localhost:9092 --topic test--partition testPartition --offset yourOffset --max-messages 10

If you don't want to stick to the binaries, I would suggest you to use kt which is a Kafka command line tool with more options and functionality.

---

For more details refer to the article How to fetch specific messages in Apache Kafka

Answer 2

Without specifying an offset and partition, you'll only be able to consume next N or first N. To consume in the "middle" of the unbounded stream, you need to give the offset

Other than console consumer, there's kafkacat

First twenty

kafkacat -C -b -t topic -o earliest -c 20

And from previous twenty (from partition zero)

kafkacat -C -b -t topic -P 0 -o -20