How to submit kafka offset after processing batch records

Question

I'm using spring-kafka and consuming batch records from kafka topic and submitting offset by AbstractMessageListenerContainer.AckMode.BATCH .

In my case processing batch records is taking time (approximately 20 seconds), and consumer thread waits until batch process is completed and then do the poll again(submits offset at this poll). In this case i will assign List of records to a thread (name: ProcessThread ) that will process all the records and give result back to consumer thread, and then consumer thread will log the result. (in all this process consumer thread will wait until it gets the results from the ProcessThread , which causing low performance.

Is there any way that ProcessThread will take care of submitting offset to kafka? so that consumer thread will not need to wait, and for every poll it will create a new processThread

In my case i have topic with 20 partitions and 10 pods, each pod with 2 consumer threads (spring kafka concurrent consumers), each poll 100 records (processing all these records by using spring boot threading @Async )

By above configuration i can process 1 million records in 2 hours and i need to drag it down to at least 40 minutes.

Any help is Appreciated 😊

Answer 1

You should upgrade to at least 1.3.7. The current version is 2.1.10. There is no separate thread in those. Only the consumer thread can send the offsets . The consumer is not thread safe.

You should not hand off to a separate thread. Use higher concurrency and more partitions.