如何从Kafka主题中检索最新消息

Question

I'm just a newbie of Kafka and I have a problem:

I have topic 'A' in Kafka, I start Spring boot app and use MessageChannel to send some message to topic 'A', then I stop app.

Is there anyway to get the latest message I've sent to topic 'A' (not all message) when I start app again? I've searched all solutions but they did not help me much, it always receive message immediately if I only sending new message. If you have runable code, pls share, I very much appreciated :(

    // Start application


    // Get latest message in topic 'A' then do some LOGIC
    if (exist latest message) {
          //Print latest message
    }

Answer 1

Your consumers store their offset (that is, the last read position). After restart, they continue reading from this point. This behavior is by design.

The offset reset properties are used when the offset for this particular consumer group is not yet known for some reason (say, it's a new consumer group or offsets have expired), but still, there are generally two options - a) reread all from the beginning b) start listening for new messages, forgetting everything that was before now.

There are hacky ways to achieve what you've described, but they are not straightforward and not recommended (a trivial one: consumer messages while just skipping them until you reach the partition EOF)

Maybe just Kafka is not the right tool for this problem.

Answer 2

One option could be combining what @saabeilin said about "re-reading all from the beggining" plus Log Compaction .

If you configure it properly, the logs will be compacted keeping only last ids, so the re-read operation will result in getting only last message sended.