如何确保邮件到达kafka经纪人？

Question

I have have a message producer on my local machine and a broker on remote host (aws).

After sending a message from the producer, I wait and call the console consumer on the remote host and see excessive logs. Without the value from producer.

The producer flushes the data after calling the send method. Everything is configured correctly.

How can I check to see that the broker received the message from the producer and to see if the producer received the answer?

Answer 1

The Send method asynchronously sends the message to the topic and returns a Future of RecordMetadata .

java.util.concurrent.Future<RecordMetadata> send(ProducerRecord<K,V> record)

Asynchronously sends a record to a topic

After the flush call, check to see that the Future has completed by calling the isDone method. (for example, Future.isDone() == true )

Invoking this method makes all buffered records immediately available to send (even if linger.ms is greater than 0) and blocks on the completion of the requests associated with these records. The post-condition of flush() is that any previously sent record will have completed (eg Future.isDone() == true). A request is considered completed when it is successfully acknowledged according to the acks configuration you have specified or else it results in an error.

The RecordMetadata contains the offset and the partition

public int partition()

The partition the record was sent to

public long offset()

the offset of the record, or -1 if {hasOffset()} returns false.

Or you can also use Callback function to ensure messages was sent to topic or not

Fully non-blocking usage can make use of the Callback parameter to provide a callback that will be invoked when the request is complete.

here is clear example in docs

 ProducerRecord<byte[],byte[]> record = new ProducerRecord<byte[],byte[]>("the-topic", key, value);
 producer.send(myRecord,
           new Callback() {
               public void onCompletion(RecordMetadata metadata, Exception e) {
                   if(e != null) {
                      e.printStackTrace();
                   } else {
                      System.out.println("The offset of the record we just sent is: " + metadata.offset());
                   }
               }
           });

Answer 2

You can try get() API of send , which will return the Future of RecordMetadata

ProducerRecord<String, String> record = 
new ProducerRecord<>("SampleTopic", "SampleKey", "SampleValue");

try {
    producer.send(record).get(); 
} catch (Exception e) {
    e.printStackTrace(); 
}

Answer 3

使用“一次发送”，您无需担心消息是否到达： https : //www.baeldung.com/kafka-exactly-once，https : //www.confluent.io/blog/一旦语义完全可能，这就是Apache Kafka的处理方式/