[英]Remote Kafka Connection Issues - Broker may not be available
I'm learning Kafka and have made the leap to using Maven.我正在学习 Kafka,并且已经开始使用 Maven。
I have a standalone Kafka instance in AWS and a Maven application on my laptop.我在 AWS 中有一个独立的 Kafka 实例,在我的笔记本电脑上有一个 Maven 应用程序。 I've written a small application which acts as a producer
我写了一个作为生产者的小应用程序
import org.apache.kafka.clients.KafkaClient;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class ProducerDemo {
public static void main(String[] args) {
// create producer properties
Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "<IP_TO_REMOTE_SERVER>:9092");
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());
//create producer
KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);
//producer record
ProducerRecord <String,String> record = new ProducerRecord<String, String>("first_topic", "jello there");
System.out.println("SENDING RECORD");
//send data - async
producer.send(record);
producer.flush();
producer.close();
System.out.println("complete");
}
}
When I run this, it appears as though I can't connect to the remote instance.当我运行它时,似乎我无法连接到远程实例。 I get the error below.
我收到以下错误。
[kafka-producer-network-thread |> producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 0 (/xx.xx.xx.xx:9092) could not be established.
[kafka-producer-network-thread |> producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] 连接到节点 0 (/xx.xx.xx.xx:9092) 可以不成立。 Broker may not be available.
经纪人可能不可用。
[main] INFO org.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
[main] INFO org.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] 使用 timeoutMillis = 9223372036854775807 ms 关闭 Kafka 生产者。
After looking at Stackoverflow, I updated the server.properties listeners section to be the private IP of the server查看 Stackoverflow 后,我将 server.properties listeners 部分更新为服务器的私有 IP
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://10.0.1.51:9092
How should I configure Kafka on the server to be accessible and listening remotely?我应该如何将服务器上的 Kafka 配置为可远程访问和侦听?
I suppose the main problem you are facing is from the configuration standpoint.我想您面临的主要问题是从配置的角度来看。 Please check if you have made all the necessary changes before communicating through producer.
在通过生产者沟通之前,请检查您是否进行了所有必要的更改。 You need to make following changes:
您需要进行以下更改:
Kafka change: You need to add configuration in Zookeeper.properties for relevant brokers. Kafka 更改:您需要在 Zookeeper.properties 中为相关代理添加配置。
AWS change: While connecting to AWS you need to set up way to pass.pem file. AWS 更改:连接到 AWS 时,您需要设置 pass.pem 文件的方式。 You might need to enable direct access in AWS instance.
您可能需要在 AWS 实例中启用直接访问。 By default it will block all the unknown traffic.
默认情况下,它将阻止所有未知流量。
Other approach: I would recommend creating a Certificate and Key file which will whitelist your own PC as valid source.其他方法:我建议创建一个证书和密钥文件,将您自己的 PC 列入白名单作为有效来源。 Add that cert to keystore and truststore on AWS server instance.
将该证书添加到 AWS 服务器实例上的密钥库和信任库。 Change the server.properties
listeners = SSL://your.host.name:9092
& your BOOTSTRAP_SERVERS_CONFIG更改 server.properties
listeners = SSL://your.host.name:9092
& your BOOTSTRAP_SERVERS_CONFIG
Seeing the responses got me thinking about to change my config to make this work.看到回复让我考虑更改我的配置以使其正常工作。 I found a really good blog article addressing this issue here .
我在这里找到了一篇非常好的博客文章来解决这个问题。
My set up我的设置
I would stress this is not production .我要强调这不是生产。
A singular AWS EC2 instance in a VPC in a public subnet.公共子网中 VPC 中的单个 AWS EC2 实例。 Kafka installed.
卡夫卡安装。 I am connecting to Kafka as a producer from my laptop remotely using Maven.
我正在使用 Maven 从我的笔记本电脑远程连接到 Kafka 作为生产者。
No changes to zookeeper.properties zookeeper.properties没有变化
Updated server.properties , specifically the listeners and advertised.listeners更新server.properties ,特别是listeners和Advertisementd.listeners
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://<PRIVATE_IP_ADDRESS>:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://<PUBLIC_IP_ADDRESS>:9092
Then in my Maven code, for the BOOTSTRAP_SERVERS_CONFIG I reference the public IP然后在我的 Maven 代码中,对于BOOTSTRAP_SERVERS_CONFIG我引用公共 IP
import org.apache.kafka.clients.KafkaClient;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class ProducerDemo {
public static void main(String[] args) {
// create producer properties
Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "<PUBLIC_IP_ADRESS>:9092");
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());
//create producer
KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);
//producer record
ProducerRecord <String,String> record = new ProducerRecord<String, String>("first_topic", "good pony");
System.out.println("SENDING RECORD");
//send data - async
producer.send(record);
producer.flush();
producer.close();
System.out.println("complete");
}
}
This runs successfully这运行成功
SENDING RECORD
发送记录
[kafka-producer-network-thread |
[kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-1]
producer-1] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-1]
[main] INFOorg.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
[main] INFOorg.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] 使用 timeoutMillis = 9223372036854775807 ms 关闭 Kafka 生产者。
complete
完全的
We see the text pushed to the consumer我们看到推送给消费者的文字
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.