無法將第二個 kafka 實例 dockerize 到 zookeeper

Question

我想用兩個 kafka 實例對一個 kafka 集群進行 dockerize。

運行動物園管理員

docker run -it --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper

運行卡夫卡

docker run -it --name kafka -p 9092:9092 -e ADVERTISED_HOST_NAME=$(hostname -f) --link zookeeper:zookeeper debezium/kafka

運行一個 zookeeper 和一個 kafka 是成功的，但是添加了第二個 kafka 容器，如下所示

docker run -it --name kafka2 -p 9096:9092 -e ADVERTISED_HOST_NAME=$(hostname -f) --link zookeeper:zookeeper debezium/kafka

給我以下錯誤：

2019-09-24 04:11:48,728 - ERROR [main:Logging@74] - Error while creating ephemeral at /brokers/ids/1, node already exists and owner '72057611679825940' does not match current session '72057611679825967'
2019-09-24 04:11:48,748 - ERROR [main:MarkerIgnoringBase@159] - [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
    at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1784)
    at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1722)
    at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1689)
    at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:97)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:260)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:75)
    at kafka.Kafka.main(Kafka.scala)

Answer 1

在 debezium Kafka 容器中，您需要傳遞-e BROKER_ID=2 。 默認值為 1。

https://github.com/debezium/docker-images/blob/master/kafka/0.10/docker-entrypoint.sh#L6-L9

---

在其他 Kafka 容器中，您可以使用一個KAFKA_BROKER_ID變量，該變量需要為每個容器設置不同的變量，該變量還將內部主題的復制因子設置為 2 並正確設置廣告偵聽器。

這是一個示例撰寫文件

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka-1:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 19092:19092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_HOST://localhost:19092,PLAINTEXT://kafka-1:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2

  kafka-2:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_HOST://localhost:29092,PLAINTEXT://kafka-2:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2

前任。

$ kafkacat -L -b localhost:19092
Metadata for all topics (from broker 1: localhost:19092/1):
 2 brokers:
  broker 2 at localhost:29092
  broker 1 at localhost:19092 (controller)
 1 topics:
  topic "__confluent.support.metrics" with 1 partitions:
    partition 0, leader 1, replicas: 1,2, isrs: 1,2

注意：在一台機器上運行多個代理不會提高 Kafka 的任何性能或可靠性