Spring Kafka KafkaTemplate.flush() required?

Question

I am using Spring kafka for the first time and I have created a Producer and Consumer by using spring kafka. I have my kafka server running on localhost and have created a topic called test. I was not able to send messages to the consumer by simply calling

        KafkaTemplate.send(topicName,Data);

I had to call flush() on kafkaTemplate after calling send on the same object and then the consumer was able to receive the data. Okay it works and it is fantastic. But could anyone explain to me what is happening behind the scenes? Why is the flush method required to be called.

From the official spring kafka documentation.

           public void flush()

Flush the producer. Note It only makes sense to invoke this method if the ProducerFactory serves up a singleton producer (such as the DefaultKafkaProducerFactory).

Thank you in advance.

Answer 1

the implement of producer is async. Message is stored in an internal queue to wait to send by inner thread, which would improve efficiency with potential batching.

So, messages may stay in client's memory when program exit. In this case, Kafka server don't actually receive these messages.

message would be sent in timeout defined by queue.buffering.max.ms , or other size / number limit.

flush force all message in send queue to be delivered to server.