Delayed ACK in Spring Kafka

Question

I'm using Spring and Spring Kafka for a batching service that collects data from Kafka until certain conditions are met, and then dumps the data.

I want to acknowledge the commits when the data leaves my service, but it could potentially sit in memory for 5-10 minutes.

Given that the Acknowledgement implementations in Spring Kafka hold on to the original record(s) it seems unreasonable to hold on to them until I dump my data given what that would do to memory utilization.

Is there any other way to acknowledge / commit offsets from Spring Kafka given just the partition/offset information?

Answer 1

You could use AckMode.TIME or AckMode.COUNT with an incredibly large ackTime or ackCount so the container will never do the ack.

Then, pass the Consumer<?, ?> into your listener method and do the offset commit yourself.

Note, however, that the consumer is not thread-safe so you must perform the commit on the listener thread.

Also, bear in mind that records are not individually ack'd, just the offset. You can't ack "out of order".

Also, you would likely need to increase the max.poll.interval.ms beyond its default (5 minutes) to avoid a rebalance of the partitions.