[英]How can I produce messages with Kafka 8.2 API in Java?
I'm trying to work with the kafka API in java. 我正在尝试使用java中的kafka API。 I'm using the following maven dependency:
我正在使用以下maven依赖项:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.8.2.0</version>
</dependency>
I'm having trouble connecting to a remote kafka server. 我无法连接到远程kafka服务器。 I changed the kafka 'server.properties' file port attribute to be port 8080. I can start both the zookeeper and the kafka server no problem.
我将kafka'server.properties'文件端口属性更改为端口8080.我可以启动zookeeper和kafka服务器没问题。 I can also use the console producer and consumer applications that came with the kafka download.
我还可以使用kafka下载附带的控制台生产者和消费者应用程序。 (Scala 2.10 version)
(Scala 2.10版)
I'm using the following client code to create a remote KafkaProducer 我正在使用以下客户端代码来创建远程KafkaProducer
Properties propsProducer = new Properties();
propsProducer.put("bootstrap.servers", "172.xx.xx.xxx:8080");
propsProducer.put("key.serializer", org.apache.kafka.common.serialization.ByteArraySerializer.class);
propsProducer.put("value.serializer", org.apache.kafka.common.serialization.ByteArraySerializer.class);
propsProducer.put("topic.metadata.refresh.interval.ms", "0");
KafkaProducer<byte[], byte[]> m_kafkaProducer = new KafkaProducer<byte[], byte[]>(propsProducer);
Once I've created the producer, I can run the following line and get valid topic info returned, granted strTopic is an existing topic name. 一旦我创建了生成器,我就可以运行以下行并返回有效的主题信息,授予strTopic是现有的主题名称。
List<PartitionInfo> partitionInfo = m_kafkaProducer.partitionsFor(strTopic);
When I try to send a message, I do the following: 当我尝试发送消息时,我执行以下操作:
ProducerRecord<byte[], byte[]> prMessage = new ProducerRecord<byte[],byte[]>(strTopic, strMessage.getBytes());
RecordMetadata futureData = m_kafkaProducer.send(prMessage).get();
The call to send() blocks indefinitely and when I manually terminate the process, I see that the ERROR Closing socket because of error on kafka server(IOException, Connection Reset by Peer) error. 无限期地调用send()块,当我手动终止进程时,由于kafka服务器上的错误(IOException,Peer连接重置)错误,我看到ERROR Closing socket。
Also, it's worth nothing that the host.name, advertised.host.name, and advertised.port properties are all still commented out on the 'server.properties' file. 此外,在“server.properties”文件中仍然注释掉host.name,advertised.host.name和advertised.port属性并不值得。 Oh, and if I change the line:
哦,如果我换行:
propsProducer.put("bootstrap.servers", "172.xx.xx.xxx:8080");
to 至
propsProducer.put("bootstrap.servers", "127.0.0.1:8080");
and run it on the same server as where the kafka server is installed, it works but I'm trying to work with it remotely. 并在与安装kafka服务器的服务器相同的服务器上运行它,但是我正在尝试远程使用它。
Appreciate any help and if I can clarify at all let me know. 感谢任何帮助,如果我能澄清一下,请告诉我。
After lots of digging, I decided to implement the example found here: Kafka Producer Example . 经过大量的挖掘,我决定实现这里的例子: Kafka Producer示例 。 I shortened the code and didn't implement a partitioner class.
我缩短了代码并没有实现分区器类。 I updated my pom with the dependency listed and I was still having the same issue.
我用列出的依赖项更新了我的pom,但我仍然遇到同样的问题。 Ultimately, I made some configuration changes and everything worked.
最终,我做了一些配置更改,一切正常。
The final piece of the puzzle was defining the Kafka server in /etc/hosts of both the server and the client machines. 最后一个难题是在服务器和客户机的/ etc / hosts中定义Kafka服务器。 I added the following to both files.
我在两个文件中添加了以下内容。
172.xx.xx.xxx serverHost1
Again, the x's are just masks. 再次,x只是面具。 Then, I set the advertised.host.name in the server.properties file to serverHost1.
然后,我将server.properties文件中的advertised.host.name设置为serverHost1。 NOTE: I got that IP after running an ifconfig on the server machine.
注意:在服务器计算机上运行ifconfig后,我获得了该IP。
I changed the line 我换了线
propsProducer.put("metadata.broker.list", "172.xx.xx.xxx:8080");
to 至
propsProducer.put("metadata.broker.list", "serverHost1:8080");
The Kafka API didn't like the fact that I was defining an IP as a string. Kafka API不喜欢我将IP定义为字符串的事实。 Instead it was looking up the IP from within the etc/hosts file although the documentation says:
相反,它是在etc / hosts文件中查找IP,尽管文档说:
"Hostname the broker will advertise to producers and consumers. If not set, it uses the value for "host.name" if configured. Otherwise, it will use the value returned from java.net.InetAddress.getCanonicalHostName()." “代理将向生产者和使用者通告主机名。如果未设置,则使用”host.name“的值(如果已配置)。否则,它将使用从java.net.InetAddress.getCanonicalHostName()返回的值。”
Which will just return the IP, in the string form, I was previously using if not defined in etc/hosts of client machine, otherwise it returns the name paired with the IP (serverHost1 in my case). 哪个将以字符串形式返回IP,如果未在客户端机器的etc / hosts中定义,我之前使用,否则返回与IP配对的名称(在我的情况下为serverHost1)。 Also, I never did set the value of host.name either.
另外,我从未设置过host.name的值。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.