[英]whether apache Kafka 3.X cluster requires already started zookeeper or not
[英]Apache Kafka cluster not connecting to Zookeeper on Docker
我正在 Docker 上创建一个 3 节点 Zookeeper 和 3 节点 Kafka 集群。 我需要将 Kafka 集群链接到 Zookeeper 集群。
下面是 docker-compose.yml 代码 -
version: '2'
services:
zookeeper-1:
image: zookeeper:latest
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: 22181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 5
ZOOKEEPER_SYNC_LIMIT: 2
ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
zookeeper-2:
image: zookeeper:latest
environment:
ZOOKEEPER_SERVER_ID: 2
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 5
ZOOKEEPER_SYNC_LIMIT: 2
ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
zookeeper-3:
image: zookeeper:latest
environment:
ZOOKEEPER_SERVER_ID: 3
ZOOKEEPER_CLIENT_PORT: 42181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 5
ZOOKEEPER_SYNC_LIMIT: 2
ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
kafka1:
image: abc/kafka:latest
hostname: kafka1
network_mode: host
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:19092
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
kafka2:
image: abc/kafka:latest
hostname: kafka2
network_mode: host
environment:
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
kafka3:
image: abc/kafka:latest
hostname: kafka3
network_mode: host
environment:
KAFKA_BROKER_ID: 3
KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:39092
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
当我运行 docker-compose up 命令时,3 节点 zookeeper 集群创建成功。 但是没有创建 Kafka 集群。 它只在命令行上显示以下错误 -
kafka_kafka2_1 exited with code 0
kafka_kafka1_1 exited with code 0
kafka_kafka3_1 exited with code 0
我还添加了用于创建图像的 Dockerfile,如下所示 -
FROM centos:7
MAINTAINER abc@xyz.com
ENV KAFKA_BIN=http://www-eu.apache.org/dist/kafka/0.11.0.2/kafka_2.11-0.11.0.2.tgz
RUN yum install -y wget java-1.8.0-openjdk \
&& cd /tmp && wget -q $KAFKA_BIN \
&& export K_TAR=/tmp/$(ls kafka* | head -1) \
&& mkdir -p /opt/apache/kafka/ && tar -zxf $K_TAR -C /opt/apache/kafka/ \
&& cd /opt/apache/kafka && ln -s $(ls) current \
&& rm -rf $K_TAR
ENV KAFKA_HOME /opt/apache/kafka/current
ENV PATH $PATH:$KAFKA_HOME/bin
ADD resources /home/kafka
RUN groupadd -r kafka \
&& useradd -r -g kafka kafka \
&& mkdir -p /home/kafka \
&& chown -R kafka:kafka /home/kafka \
&& chmod -R +x /home/kafka/scripts \
&& mkdir -p /var/log/kafka \
&& chown -R kafka:kafka /var/log/kafka \
&& mkdir -p /etc/kafka \
&& chown -R kafka:kafka /etc/kafka
USER kafka
没有任何类型的错误消息。 有人可以帮我调试这个问题,以了解为什么 kafka 集群没有启动。 我无法调试这个,特别是因为它只以代码 0 退出并且没有提供错误消息。
首先,您忘记公开zookeeper 端口。 其次,localhost 意味着 localhost——即使对于 docker 容器也是如此。 所以你的 Kafka 容器会尝试连接到同一个容器内的 zookeeper,这是行不通的。
看看这里是如何正确完成的: https : //github.com/confluentinc/cp-docker-images/blob/4.0.x/examples/cp-all-in-one/docker-compose.yml
如果您不喜欢这种方法,则可以在不公开任何端口的情况下进行。 为此,请在撰写文件的 zookeeper 环境部分设置 zookeeper 客户端端口:
...
services:
zookeeper:
...
environment:
ZOOKEEPER_CLIENT_PORT: 2181
其次,将您的 kafka 容器连接到它:
...
kafka:
...
environment:
ZOOKEEPER_SERVERS: zookeeper:2181
使用这种方法,一切都将发生在 docker compose 网络中。
Docker 中的localhost
指的是容器实例,而不是主机。 这是 Kafka 安装混乱的主要来源。 您需要配置 Kafka 的侦听器以通告不同的主机:端口组合 — 可从容器内访问的组合。
下面显示了如何为多个代理完成此操作的示例。 信用:该示例取自Effective Kafka随附的存储库。
version: "3.2"
services:
zookeeper:
image: bitnami/zookeeper:3
ports:
- 2181:2181
environment:
ALLOW_ANONYMOUS_LOGIN: "yes"
kafka-0:
image: bitnami/kafka:2
ports:
- 9092:9092
environment:
KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
ALLOW_PLAINTEXT_LISTENER: "yes"
KAFKA_LISTENERS: >-
INTERNAL://:29092,EXTERNAL://:9092
KAFKA_ADVERTISED_LISTENERS: >-
INTERNAL://kafka-0:29092,EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >-
INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
depends_on:
- zookeeper
kafka-1:
image: bitnami/kafka:2
ports:
- 9093:9093
environment:
KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
ALLOW_PLAINTEXT_LISTENER: "yes"
KAFKA_LISTENERS: >-
INTERNAL://:29092,EXTERNAL://:9093
KAFKA_ADVERTISED_LISTENERS: >-
INTERNAL://kafka-1:29092,EXTERNAL://localhost:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >-
INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
depends_on:
- zookeeper
kafka-2:
image: bitnami/kafka:2
ports:
- 9094:9094
environment:
KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
ALLOW_PLAINTEXT_LISTENER: "yes"
KAFKA_LISTENERS: >-
INTERNAL://:29092,EXTERNAL://:9094
KAFKA_ADVERTISED_LISTENERS: >-
INTERNAL://kafka-2:29092,EXTERNAL://localhost:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >-
INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
depends_on:
- zookeeper
kafdrop:
image: obsidiandynamics/kafdrop:latest
ports:
- 9000:9000
environment:
KAFKA_BROKERCONNECT: >-
kafka-0:29092,kafka-1:29092,kafka-2:29092
depends_on:
- kafka-0
- kafka-1
- kafka-2
此处,连接到代理的应用程序是 Kafdrop,但您可以将其替换为您自己的映像。 此外,为简洁起见,该示例仅使用一个 ZK 节点和 3 个代理; 这应该是一个直接的改变。
您需要在/etc/hosts
设置正确的主机名,以便能够连接 docker 之外的 kafka 容器:
cat /etc/hosts
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1 localhost
127.0.0.1 kafka1
127.0.0.1 kafka2
127.0.0.1 kafka3
这是我的工作配置:
version: '2'
services:
zookeeper1:
image: confluentinc/cp-zookeeper:$CP_VERSION
container_name: zookeeper1
hostname: zookeeper1
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_SERVERS: "zookeeper1:22888:23888;zookeeper2:22888:23888;zookeeper3:22888:23888"
ports:
- 2181:2181
volumes:
- ./data/zoo-1/data:/var/lib/zookeeper/data
- ./data/zoo-1/log:/var/lib/zookeeper/log
zookeeper2:
image: confluentinc/cp-zookeeper:$CP_VERSION
container_name: zookeeper2
hostname: zookeeper2
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_SERVER_ID: 2
ZOOKEEPER_SERVERS: "zookeeper1:22888:23888;zookeeper2:22888:23888;zookeeper3:22888:23888"
ports:
- 2182:2181
volumes:
- ./data/zoo-2/data:/var/lib/zookeeper/data
- ./data/zoo-2/log:/var/lib/zookeeper/log
zookeeper3:
image: confluentinc/cp-zookeeper:$CP_VERSION
container_name: zookeeper3
hostname: zookeeper3
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_SERVER_ID: 3
ZOOKEEPER_SERVERS: "zookeeper1:22888:23888;zookeeper2:22888:23888;zookeeper3:22888:23888"
ports:
- 2183:2181
volumes:
- ./data/zoo-3/data:/var/lib/zookeeper/data
- ./data/zoo-3/log:/var/lib/zookeeper/log
kafka1:
image: confluentinc/cp-kafka:$CP_VERSION
container_name: kafka1
hostname: kafka1
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
ports:
- 9091:9091
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INTERNAL://:29092,OUTSIDE://:9091
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,OUTSIDE://kafka1:9091
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_LOG_DIRS: /var/lib/kafka/logs
kafka2:
image: confluentinc/cp-kafka:$CP_VERSION
container_name: kafka2
hostname: kafka2
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INTERNAL://:29092,OUTSIDE://:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:29092,OUTSIDE://kafka2:9092
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_LOG_DIRS: /var/lib/kafka/logs
kafka3:
image: confluentinc/cp-kafka:$CP_VERSION
container_name: kafka3
hostname: kafka3
depends_on:
- zookeeper1
- zookeeper2
- zookeeper3
ports:
- 9093:9093
environment:
KAFKA_BROKER_ID: 3
KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INTERNAL://:29092,OUTSIDE://:9093
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:29092,OUTSIDE://kafka3:9093
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_LOG_DIRS: /var/lib/kafka/logs
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.