将kafka消息归档到AWS S3时如何实现“精确一次”语义？

Question

How to store kafka message data with partition offset in one S3 PutObject transaction to achieve Exactly Once semantics? Does it possible?

Answer 1

Yes it should be possible. One way to do that is take control of the offset management.

Your consumer can read one message from Kafka at at time and put that as object in AWS, while doing that store offset + partition name as key in the AWS. Now lets say your client crashed. When it comes up next time you query S3 to find what is the last offset in S3 and start reading message from there. For additional protection before you put message in S3 check if object with that key (It would be better if your producer produces UUID for message and you can use that) exists in S3 if yes dont overwrite it instead skip the message.

kafkaConsumer.subscribe(Arrays.asList(topicName), new ConsumerRebalanceListener() {
     public void onPartitionsRevoked(Collection<TopicPartition> partitions) {}
     public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
        Iterator<TopicPartition> topicPartitionIterator = partitions.iterator();
        while(topicPartitionIterator.hasNext()){
              TopicPartition topicPartition = topicPartitionIterator.next();
              System.out.println("Current offset is " + kafkaConsumer.position(topicPartition) + " committed offset is kafkaConsumer.committed(topicPartition) 
              System.out.println("Resetting offset to " + startingOffset);
              kafkaConsumer.seek(topicPartition, startingOffset);
           }
        }
      }
  });

Hope that helps