[英]Unable to run console consumer using a Kafka broker inside a Docker container
I'm having a problem running a Kafka broker in a Docker container.我在 Docker 容器中运行 Kafka 代理时遇到问题。
I have downloaded and unpacked a tar archive of Kafka 2.12-2.4.1.我已经下载并解压了 Kafka 2.12-2.4.1 的 tar 存档。
I can run Zookeeper and Kafka brokers from the command line, producing and consuming messages successfully.我可以从命令行运行 Zookeeper 和 Kafka 代理,成功地生产和消费消息。 When I build a docker container with Zookeeper, I can run that with a Kafka broker running from the command line, again producing and consuming messages successfully.
当我使用 Zookeeper 构建 docker 容器时,我可以使用从命令行运行的 Kafka 代理运行它,再次成功生成和使用消息。 When I put the broker into a docker container, the consumers and producers can't find a broker to connect to.
当我将代理放入 docker 容器时,消费者和生产者找不到要连接的代理。
I have checked to ensure that Zookeeper has a connected broker: the zookeeper "dump" command below我已经检查以确保 Zookeeper 有一个连接的代理:下面的 zookeeper "dump" 命令
echo dump | nc localhost 2181
returns返回
SessionTracker dump: Session Sets (3)/(1): 0 expire at Thu May 28 09:53:33 GMT 2020: 0 expire at Thu May 28 09:53:36 GMT 2020: 1 expire at Thu May 28 09:53:39 GMT 2020: 0x10002dfed700008 ephemeral nodes dump: Sessions with Ephemerals (1): 0x10002dfed700008: /controller /brokers/ids/0 Connections dump: Connections Sets (2)/(2): 0 expire at Thu May 28 09:53:37 GMT 2020: 2 expire at Thu May 28 09:53:47 GMT 2020: ip: /172.18.0.1:55656 sessionId: 0x0 ip: /172.18.0.5:48474 sessionId: 0x10002dfed700008
which, to my untrained eye, looks like Zookeeper has a registered broker.在我未经训练的眼里,这看起来就像 Zookeeper 有一个注册经纪人。
I can successfully list topics with the command我可以使用命令成功列出主题
kafka-topics.sh --list --zookeeper localhost:2181
and when I run the command当我运行命令时
netstat -plnt
I can see that I have listeners on ports 2181 and 9092, as below我可以看到我在端口 2181 和 9092 上有监听器,如下所示
tcp6 0 0 :::9092 :::* LISTEN 54661/docker-proxy tcp6 0 0 :::2181 :::* LISTEN 47872/docker-proxy
However, when I try to connect a consumer with the command但是,当我尝试使用命令连接消费者时
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic aaaa
I get errors我收到错误
WARN [Consumer clientId=consumer-console-consumer-8228-1, groupId=console-consumer-8228] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
which would seem to indicate that the client can't contact the broker.这似乎表明客户无法联系经纪人。
I've tried several different variations on the listeners and advertised.listeners in the broker config file, setting them to localhost and 127.0.0.1, but to no avail.我在代理配置文件中尝试了几种不同的监听器和广告监听器,将它们设置为 localhost 和 127.0.0.1,但无济于事。
I'm sure that I'm missing something simple: can anyone help?我确定我缺少一些简单的东西:有人可以帮忙吗?
Make sure to configure the following:确保配置以下内容:
KAFKA_LISTENERS: LISTENER_INTERNAL://kafka-host:29092,LISTENER_EXTERNAL://kafka-host:9092
KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka-host:29092,LISTENER_EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
Now all your clients that run within the Docker Network, should use现在,在 Docker 网络中运行的所有客户端都应该使用
LISTENER_INTERNAL://kafka-host:29092
All clients outside the Docker Network should use Docker 网络之外的所有客户端都应使用
LISTENER_EXTERNAL://kafka-host:9092
For a more comprehensive guide, read @RobinMoffatt article .如需更全面的指南,请阅读@RobinMoffatt 文章。
So, I have solved the problem and I now have a working Docker/Kafka environment: I have a zookeeper, a broker and a consumer running in docker, and I can produce messages both from a producer running in docker and a consumer running from a command line.所以,我已经解决了这个问题,我现在有一个工作的 Docker/Kafka 环境:我有一个 Zookeeper、一个代理和一个运行在 docker 中的消费者,我可以从运行在 docker 中的生产者和运行在命令行。
As @GiorgosMyrianthous suspected, the problem was with the listeners and advertised listeners.正如@GiorgosMyrianthous 所怀疑的那样,问题出在听众和广告听众身上。 I entered the following values in the Kafka broker properties file
我在 Kafka 代理属性文件中输入了以下值
listeners=PLAINTEXT://:29092,PLAINTEXT_HOST://:9092 advertised.listeners=PLAINTEXT://server:29092,PLAINTEXT_HOST://localhost:9092 listener.security.protocol.map=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
and I started the container with port 9092 exposed to the outside world.我启动了9092端口的容器,暴露在外面。
I ran the Docker consumer with the bootstrap server set to server:29092
.我运行 Docker 消费者,并将引导服务器设置为
server:29092
。
I ran the Docker producer with the bootstrap server set to server:29092
.我运行 Docker 生产者,引导服务器设置为
server:29092
。
I ran the command line producer with the bootstrap server set to localhost:9092
.我在引导服务器设置为
localhost:9092
的情况下运行了命令行生产者。
Messages typed in either of the producers appeared in the consumer.在任一生产者中键入的消息出现在消费者中。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.