Kafka：如何显示偏移量

Question

I'm super new to Kafka. I've installed kafka and zookeeper using homebrew on my mac, and I'm playing around with the quickstart guide .

I've been able to push messages onto Kafka using the following command and STDIN

kafka-console-producer --broker-list localhost:9092 --topic test

and I can read things off using

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

What's not clear to me is how I use offsets. It's my understanding that each message added to a topic will have a numerical, incremental offset value. However, if I try to do something like this

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset 1

I get a non-zero status code and shows me no messages (other than the usual help/usage information)

I also can't get the latest or earliest keywords to work

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset earliest
kafka-console-consumer --bootstrap-server localhost:9092 --topic test --offset latest

Both of the above also return non-zero status codes.

Do I fundamentally misunderstand offsets? If not, is there a way to list all messages with their offsets? Finally -- what's the simpliest example of the --offset flag for the kafka-console-consumer ?

Answer 1

If you looked at the output after you gave the offset value, it would have said that you needed to specify a partition (at the top of the help section)

Topics are subdivided into partitions, and an offset of 1 could only exist on one partition out of potentially hundreds, so you must specify it

Regarding displaying the offsets , lookup the GetOffsetShell command syntax

Answer 2

A nice command line tool that can display the offset with each message is kafkacat .

kafkacat -b localhost:9092 -C -t test -f 'Topic %t [%p] at offset %o: key %k: %s\n'

It will print out something like

Topic test [5] at offset 111: key "0171bf8102007900e33": {"Message": "1"} 
Topic test [2] at offset 123: key "070021b0f001f614c1b": {"Message": "2"}

Answer 3

Since GetOffsetShell only works with PLAINTEXT , it can be inconvenient for many.
Good news is that additional properties, including print.offset seems to have made their way into 2.7 based on the communication in 9099 PR .
It must be possible to use print.offset=true now!

Answer 4

使用offset时需要分区

.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --partition 0 --offset 0 --property print.key=true --property print.timestamp=true

Answer 5

You have to give partition value as well with offset in above command like this

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --partition 0 --offset 1