[英]Kafka - Producer Number of Batches
Is there a way to determine the number of batches that were created by a Kafka producer for a specific set of messages? 有没有一种方法可以确定Kafka生产者为一组特定的消息创建的批次数量? For instance if I am sending 10K messages in a loop , is there a way to check how many batches were sent?
例如,如果我循环发送10K消息,是否可以检查发送了多少批? I set the "batch.size" to a high value and my expectation was that the message will be buffered and there will be a delay in seeing the message in my consumer.
我将“ batch.size”设置为一个很高的值,我期望消息将被缓冲,并且在我的使用者中看到该消息会有一个延迟。 However this seems to be printed almost immediately in my consumer program.
但是,这似乎几乎在我的客户程序中立即打印。
The default value if batch.size is 16384. Is this number of bytes? 如果batch.size的默认值为16384。这是字节数吗?
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
public class KafkaProducerApp {
public static void main(String[] args){
Properties properties = new Properties();
properties.put("bootstrap.servers","localhost:9092,localhost:9093,localhost:9094");
properties.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer");
properties.put("value.serializer","org.apache.kafka.common.serialization.StringSerializer");
properties.put("acks","0");
properties.put("batch.size",33554432);
KafkaProducer<String,String> kafkaProducer = new KafkaProducer<String, String>(properties);
Map<Integer,Integer> partitionCount = new HashMap<Integer,Integer>();
partitionCount.put(0,0);
partitionCount.put(1,0);
partitionCount.put(2,0);
try{
Date from = new Date();
for(int i=0;i<10000;i++) {
RecordMetadata ack = kafkaProducer.send(new ProducerRecord<String, String>("test_topic", Integer.toString(i), "MyMessage" + Integer.toString(i))).get();
//RecordMetadata ack = kafkaProducer.send(new ProducerRecord<String,String>("test_topic",0,Integer.toString(i), "MyMessage" + Integer.toString(i))).get();
System.out.println(" Offset = " + ack.offset());
System.out.println(" Partition = " + ack.partition());
partitionCount.put(ack.partition(),partitionCount.get(ack.partition())+1);
}
Date to = new Date();
System.out.println(" partition 0 =" + partitionCount.get(0));
System.out.println(" partition 1 =" + partitionCount.get(1));
System.out.println(" partition 2 =" + partitionCount.get(2));
System.out.println(" Elapsed Time = " + (to.getTime()-from.getTime())/1000);
} catch (Exception ex){
ex.printStackTrace();
} finally {
kafkaProducer.close();
}
}
}
What you are asking for is the total number of produce requests. 您要的是生产请求的总数。
You can see the average number of produce requests per second using the JMX Mbean kafka.producer:type=producer-metrics,client-id=([-.w]+) 您可以使用JMX Mbean kafka查看每秒平均生产请求数。producer:type = producer-metrics,client-id =([-。w] +)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.