How to get only the latest message of a Kafka topic?

Question

Is there a way to decompose the messages so that only the latest message is consumed?

I tried to save the messages in a list, but it didn't really work out

var consumer = new Consumer(new ConsumerOptions(topic, router));

foreach (var message in consumer.Consume())
{
    Console.WriteLine(Encoding.UTF8.GetString(message.Value));
}

Output should be: 1, 2, 3, 4

Output is: 1, 1, 2, 1, 2, 3, 1, 2, 3, 4

Answer 1

As a matter of fact, the only way to do what you want is to have a topic with single partition and setting max.poll.records to one.

Otherwise, there is no way to to it, because the last message does not make sense. Any input message can be pushed into different partitions and you have some last messages relative to the number of partitions your topic has.

Answer 2

You can use something called Log Compaction which will "truncate" messages with the same key. So when you send messages with the same key you will get only the last message/value for that key. This feature is enabled by default. When you send the messages 1, 1, 2, 1, 2, 3, 1, 2, 3, 4 and you want to read them as 1, 2, 3, 4 you give the messages the same keys, which should be overwritten by each other (so all the 1 messages get the same key, all the 2 messages get the same key, ...).