What happens if I don't close the kafka producer

Question

I'm processing xml's and I need to send a message per record, when I receive the last record I close the kafka producer, the problem here is that the send method of the kafka producer is async, therefore, sometimes when I close the producer it trows java.lang.IllegalStateException: Cannot send after the producer is closed. I've read somewhere that I can leave the producer open. My question is: What does it imply, or if there is a better solution for this.

---Edit---

<list>
  <element attr1="" att2="" attr3=""/>
  <element attr1="" att2="" attr3=""/>
  <element attr1="" att2="" attr3=""/>
  <element attr1="" att2="" attr3=""/>
  <element attr1="" att2="" attr3=""/>
  <element attr1="" att2="" attr3=""/>
  <element attr1="" att2="" attr3=""/>
  <element attr1="" att2="" attr3=""/>
...
</list>

Imagine the following scenario:

• We read the tag and we create the kafka producer
• Per each element we read its attributes, generate a json object and send it to kafka using the send method. -When we read the element we call the close method in the producer

Problem the number of elements can be 80k therefore, sometimes when we call the disconnect method it continues sending the messages in an async way. So we need to call the flush method first but it impacts the performance

Answer 1

You should call Producer.flush() before calling Producer.close() . This is a blocking call and will return not before all record got sent.

If you don't call close() , depending on the implementation/language you might end up with resource/memory leaks.