How to send Meta data after producing all message to topic in kafka java?

Question

I am producing student details to a Kafka topic by using the Java Kafka client libraries and what I want is once all the records are produced, I want to send the message count with some other meta data to the same Kafka topic.

 List<Student> allStudent;

suppose the size of the allStudent list is 10

Then once all student records are produced to the topic then I want to produce another message like: "10 records successfully produced to topic".

"10 record successfully produced to topic" - This should be the last message to be produced.

Answer 1

The Kafka producer send() method returns a Future<RecordMetadata> object. One option would be to call the get() method on this Future. Once the get() method returns, you can be sure that the data has been written to Kafka. Also you should probably configure your producer for durability ( acks=all setting). Once your get method returns, you can send your confirmation message.

Still I would consider sending those confirmation messages a questionable practice. What are they good for?

Answer 2

I am not sure what is the real business case for this.

But in KAFKA we call this behaviour as a Control Message, which is a kind of Meta Data about the list of records previously sent for reconciliation purpose later, especially for the batch processing. But the control message is completely driven by the producer ie producer should have a counter of messages sent Synchronously or Asynchronously and send a control message to another topic, let's call it as a control topic, where a consumer subscribed to perform the delta comparison between the Data Topic (Event Schema) and Control Topic (Control Message Schema).

Along with this, if I want to have an extremely high confidence that broker leaders and replicas have the data produced, you should use acks=all with right values set for replication factor and in-sync replicas parameters.

In summary, it is up to the producer how you want to implement this as KAFKA don't have a specific implementation for this.