反序列化錯誤后如何提交偏移量？ 春天卡夫卡

Question

我能夠使用 ErrorDesrializationHandler 成功處理反序列化錯誤，但是當我重新啟動消費者時，它再次開始重新處理由於反序列化而導致的所有失敗消息。
由於反序列化異常沒有進入 Kafka Listener，如何確認並提交偏移量？
謝謝。

我正在使用的自定義錯誤處理程序：

class KafkaErrorHandler implements ConsumerAwareErrorHandler {

private static final Logger LOG = LoggerFactory.getLogger(KafkaErrorHandler.class);

@Override
public void handle(Exception thrownException, List<ConsumerRecord<?, ?>> records, Consumer<?, ?> consumer, MessageListenerContainer container) {
    doSeeks(records, consumer);
    if (!records.isEmpty()) {
        ConsumerRecord<?, ?> record = records.get(0);
        String topic = record.topic();
        long offset = record.offset();
        int partition = record.partition();
        if (thrownException.getClass().equals(DeserializationException.class)) {
            DeserializationException exception = (DeserializationException) thrownException;
            String malformedMessage = new String(exception.getData());
            LOG.info("Skipping message with topic {} and offset {} " +
                    "- malformed message: {} , exception: {}", topic, offset, malformedMessage, exception.getLocalizedMessage());
        } else {
            LOG.info("Skipping message with topic {} - offset {} - partition {} - exception {}", topic, offset, partition, thrownException);
        }
    } else {
        LOG.info("Consumer exception - cause: {}", thrownException.getMessage());
    }
}

private void doSeeks(List<ConsumerRecord<?, ?>> records, Consumer<?, ?> consumer) {
    Map<TopicPartition, Long> partitions = new LinkedHashMap<>();
    AtomicBoolean first = new AtomicBoolean(true);
    records.forEach(record ->  {
        if (first.get()) {
            partitions.put(new TopicPartition(record.topic(), record.partition()), record.offset() + 1);
        } else {
            partitions.computeIfAbsent(new TopicPartition(record.topic(), record.partition()),
                    offset -> record.offset());
        }
        first.set(false);
    });
    partitions.forEach(consumer::seek);
}}

編輯 - - - -

private void doSeeks(List<ConsumerRecord<?, ?>> records, Consumer<?, ?> consumer, MessageListenerContainer container) {
Map<TopicPartition, OffsetAndMetadata> partitions = new LinkedHashMap<>();
AtomicBoolean first = new AtomicBoolean(true);
records.forEach(record ->  {
    if (first.get()) {
        partitions.put(new TopicPartition(record.topic(), record.partition()), new OffsetAndMetadata(record.offset()+1, record.leaderEpoch(), "A"));
    } else {
        partitions.computeIfAbsent(new TopicPartition(record.topic(), record.partition()),
                offset -> new OffsetAndMetadata(record.offset(), record.leaderEpoch(), "B"));
    }
    first.set(false);
});
partitions.forEach(consumer::seek);
consumer.commitSync(partitions);}

Answer 1

使用AckMode.MANUAL_IMMEDIATE ，請參閱

/**
 * Set to true to commit the offset for a recovered record.
 * The container must be configured with
 * {@link org.springframework.kafka.listener.ContainerProperties.AckMode#MANUAL_IMMEDIATE}.
 * Whether or not the commit is sync or async depends on the container's syncCommits
 * property.
 * @param commitRecovered true to commit.
 */
@Override
public void setCommitRecovered(boolean commitRecovered) { // NOSONAR enhanced javadoc
    super.setCommitRecovered(commitRecovered);
}

在DefaultErrorHandler上。

使用AckMode.MANUAL是不可能的； 對於其他AckMode ，默認情況下，容器將提交恢復記錄的偏移量（基於isAckAfterHandle() ）。

編輯

提交偏移量的邏輯在SeekUtils.seekOrRecover中（在搜索之后）。

if (commitRecovered) {
    if (container.getContainerProperties().getAckMode().equals(AckMode.MANUAL_IMMEDIATE)) {
        ConsumerRecord<?, ?> record = records.get(0);
        Map<TopicPartition, OffsetAndMetadata> offsetToCommit = Collections.singletonMap(
                new TopicPartition(record.topic(), record.partition()),
                ListenerUtils.createOffsetAndMetadata(container, record.offset() + 1));
        if (container.getContainerProperties().isSyncCommits()) {
            consumer.commitSync(offsetToCommit, container.getContainerProperties().getSyncCommitTimeout());
        }
        else {
            OffsetCommitCallback commitCallback = container.getContainerProperties().getCommitCallback();
            if (commitCallback == null) {
                commitCallback = LOGGING_COMMIT_CALLBACK;
            }
            consumer.commitAsync(offsetToCommit, commitCallback);
        }
    }
    else {
        logger.debug(() -> "'commitRecovered' ignored, container AckMode must be MANUAL_IMMEDIATE, not "
                + container.getContainerProperties().getAckMode());
    }
}