Kafka consumer recover after failed message processing

Question

I'm working with simple kafka Consumer in one of my projects and my desired logic is when consumer failed to process some message it will commit last correctly processed message and then on next poll it will continue from failed message.

I tried to commit each message manually with following code:

public void fetchMessages() {
  ConsumerRecords<String, MyObject> messages = kafkaConsumer.poll(10000);
  for (ConsumerRecord message : messages) {
      logger.info("Reading kafka message, topic ["+kafkaTopic+"], partition ["+message.partition()+"], offset ["+message.offset()+"]");
      try {
          MyObject myObject = (MyObject) message.value();
          logger.info("Handling message," + myObject);
          handleMessage(myObject);
          commitMessage(message);
      } catch (Exception e) {
          logger.error("Error handling message");              throw e;
      }
  }
}


private void commitMessage(ConsumerRecord message) {
        long              nextOffset        = message.offset() + 1;

        TopicPartition    topicPartition    = new TopicPartition(kafkaTopic,message.partition());
        OffsetAndMetadata offsetAndMetadata = new OffsetAndMetadata(nextOffset);

        Map<TopicPartition,OffsetAndMetadata> offsetAndMetadataMap = new HashMap<>();
        offsetAndMetadataMap.put(topicPartition,offsetAndMetadata);

        logger.info("Commiting processed kafka message, topic ["+kafkaTopic+"], partition ["+message.partition()+"], next offset ["+nextOffset+"]");
        kafkaConsumer.commitSync(offsetAndMetadataMap);
}

But for example when I fetch 3 messages, each one from different partition, I handling the first one successfully and then failed to process second message, I just exit ConsumerRecord s for loop and I expect to get same 2 messages that I haven't committed yet in the next poll iteration. Instead consumer just continue to receive new messages and never come back to failed messages.

Also tried to apply seek on failed message and then exit the loop but it is working on 1 partition and do not work on many.

kafkaConsumer.seek(new TopicPartition(kafkaTopic,message.partition()),message.offset());    

Some details:

• topic has 12 partitions
• One consumer for all partitions
• Consumer executes poll loop one in minute
• enable.auto.commit: false

What is wrong with my code or with my logic?

Answer 1

I found how seek works, and on failed message I must seek all offsets for all partitions of current consumer.

private void seekAllPartitions() {
    logger.info("Processing of some kafka message was failed, seeking all partitions to last committed");
    List<PartitionInfo> partitionInfos = kafkaConsumer.partitionsFor(kafkaTopic);
    for (PartitionInfo partitionInfo : partitionInfos) {
        TopicPartition topicPartition = new TopicPartition(kafkaTopic, partitionInfo.partition());
        OffsetAndMetadata committedForPartition = kafkaConsumer.committed(topicPartition);
        if (committedForPartition != null) {
            kafkaConsumer.seek(topicPartition,committedForPartition.offset());
        }
    }
}

Null check for committedForPartition is needed when last offset of some consumer group on some partition wasn't set yet (unknown)