运行在不同计算机上的Kafka用户组可以接收唯一消息吗？

Question

To avoid redundant messages when consumer crashes and comes back up, I have disabled auto commit of offsets and manually committing them.

Now the question is if same topic is accessed by consumer processes on different machines, will they receive unique messages? Looking at it theoretically, manual committing will result into redundant messages received on different machines.

On my local machine I ran two instances of java consumer subscribing to same topic and they got repeated messages. How to tackle this thing? I am using high level consumer

Answer 1

Since Kafka's message delivery semantic is at-least-once , you should implement your own codes to guarantee exactly-once semantic in Kafka.

• At most once: Messages may be lost but are never redelivered.
• At least once: Messages are never lost but may be redelivered.
• Exactly once: this is what people actually want, each message is delivered once and only once.

From 4.6 Message Delivery Semantics in Kafka Documentation:

So what about exactly once semantics (ie the thing you actually want)? The limitation here is not actually a feature of the messaging system but rather the need to co-ordinate the consumer's position with what is actually stored as output. The classic way of achieving this would be to introduce a two-phase commit between the storage for the consumer position and the storage of the consumers output. But this can be handled more simply and generally by simply letting the consumer store its offset in the same place as its output. This is better because many of the output systems a consumer might want to write to will not support a two-phase commit. As an example of this, our Hadoop ETL that populates data in HDFS stores its offsets in HDFS with the data it reads so that it is guaranteed that either data and offsets are both updated or neither is. We follow similar patterns for many other data systems which require these stronger semantics and for which the messages do not have a primary key to allow for deduplication.

There is also a question which has a similar answer in Kafka FAQ: How do I get exactly-once messaging from Kafka?