[英]Kafka producer throws “Received unknown topic or partition error” when sending to topic created via AdminClient createTopics method
[英]Gets an error error in Kafka producer when creating topic but the topic is created on the Kafka server
我正在使用Kafka生產者10.2.1創建主題並寫入主題,當我創建主題時,出現以下錯誤,但是創建了主題:
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:774)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:494)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:440)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:360)
at kafka.AvroProducer.produce(AvroProducer.java:47)
at samples.TestMqttSource.messageReceived(TestMqttSource.java:89)
at mqtt.JsonConsumer.messageArrived(JsonConsumer.java:132)
at org.eclipse.paho.client.mqttv3.internal.CommsCallback.deliverMessage(CommsCallback.java:477)
at org.eclipse.paho.client.mqttv3.internal.CommsCallback.handleMessage(CommsCallback.java:380)
at org.eclipse.paho.client.mqttv3.internal.CommsCallback.run(CommsCallback.java:184)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
msg org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
loc org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
cause org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
excep java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
所有建議都受到高度贊賞。
您不能使用KafkaProducer來創建主題(因此,除非您以前通過其他方法(例如kafka admin shell腳本)進行過此操作,否則我不太確定如何創建該主題) 。 而是使用Kafka庫提供的AdminUtils。
我最近達到了您要滿足的兩個要求,您會驚訝地發現實現起來如此容易。 下面是一個簡單的代碼示例,向您顯示如何通過AdminUtils創建主題以及如何對其進行寫入。
class Foo {
private String TOPIC = "testingTopic";
private int NUM_OF_PARTITIONS = 10;
private int REPLICATION_FACTOR = 1;
public Foo() {
ZkClient zkClient = new ZkClient( "localhost:2181", 15000, 10000, ZKStringSerializer$.MODULE$ );
ZkUtils zkUtils = new ZkUtils( zkClient, new ZkConnection( "localhost:2181" ), false);
if ( !AdminUtils.topicExists(zkUtils, TOPIC) ) {
try {
AdminUtils.createTopic(zkUtils, TOPIC, NUM_OF_PARTITIONS, REPLICATION_FACTOR, new Properties(), Enforced$.MODULE$);
Properties producerConfig = new Properties();
producerConfig.put(ProducerConfig.BOOTSTRAP_SERVER_CONFIG, "localhost:9092");
producerConfig.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer");
producerConfig.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(producerConfig);
// This is just to show you how to write but you could be more elaborate
int i = 0;
while ( i < 11 ) {
ProducerRecord<String, String> rec = new ProducerRecord<>(TOPIC, ("This is line number " + i));
producer.send(rec);
i++;
}
producer.closer();
} catch ( AdminOperationException aoe ) {
aoe.printStackTrace();
}
}
}
}
請記住,如果要刪除主題,默認情況下在設置中將其禁用。 啟動Kafka時使用的配置文件(默認為$ {kafka_home} /config/server.properties) ,如果該文件不存在且設置為false或已注釋掉,則添加以下行:
delete.topic.enabled=true
然后,您必須重新啟動服務器,並可以通過Java或提供的命令行工具刪除主題。
NB
如代碼示例所示,關閉生產者/消費者總是一個好主意。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.