Kafka事务提交超时异常处理

Question

I am running kafka transactions on a large scale and below is the codepiece.

producer.initTransaction();
try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>(producerTopic, element));
    producer.commitTransaction();
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
    producer.close();
    canSendNext = false;
}catch (KafkaException e) {
    producer.abortTransaction();
}

properties used:

ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, STRING_SERIALIZER ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, BYTE_ARRAY_SERIALIZER ProducerConfig.TRANSACTIONAL_ID_CONFIG, UUID.randomUUID().toString() bootstrap.servers=localhost:9092 acks=all retries=1 partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner

while the commitTransaction is timing out the catch block of KafkaException runs and try to abort the transaction. Which fails with the error: ** Cannot attempt operation abortTransaction because the previous call to commitTransaction timed out and must be retried**

how to handle commit transaction timeout scenario

expecting the code to work

Answer 1

As per the documentation :

Note that this method will raise TimeoutException if the transaction cannot be committed before expiration of max.block.ms. Additionally, it will raise InterruptException if interrupted. It is safe to retry in either case, but it is not possible to attempt a different operation (such as abortTransaction) since the commit may already be in the progress of completing. If not retrying, the only option is to close the producer.

The problem here is that the kafka producer has timed out and it does not know whether the kafka broker will complete the transaction or not. So it cannot provide you abort transaction functionality as there is a chance that the transaction might have been committed by the broker even after the producer times out.

You should configure your kafka producer with a sufficiently large max.block.ms and good number of retries to handle such scenarios (Not sure why you have configured your retries to 1). Ideally you should be timing out very rarely - like when there is some issue in the.network or there is some actual issue going on in kafka brokers.

In such scenarios, it won't be possible for you to know if your last transaction was actually successful or not. You cannot do anything but close your kafka producer and create a new one.