堆栈内存溢出
  简体   繁体   English

多broker Confluent Kafka docker-compose.yml文件

[英]Multi-broker Confluent Kafka docker-compose.yml file

 原文  2021-10-12 21:31:46  9   2    docker/ apache-kafka/ docker-compose/ confluent-platform

问题描述

A while back I needed to instantiate a multi-broker Confluent Kafka stack using Confluent 's docker images and docker-compose(1) .前一段时间，我需要使用Confluent的 docker 图像和 docker docker-compose(1)来实例化一个多代理Confluent Kafka堆栈。 Although Confluent s documentation is quite excellent, it only offers a single-broker docker-compose.yml example;虽然Confluent的文档相当优秀，但它只提供了一个single-broker docker-compose.yml的例子； but not, say, a three-broker example from which one could study the pattern and generalize to any number of brokers needed.但不是，比如说，一个three-broker例子，人们可以从中研究模式并推广到所需的任何数量的经纪人。 I pieced the three-broker version myself.我自己拼了three-broker版本。

Sadly, I lost the result and had to manually re-create that again today.可悲的是，我丢失了结果，今天不得不再次手动重新创建它。 Web searches for this aren't great, so I'll memorialize the result as an answer to this question (for others). Web 对此的搜索不是很好，所以我会记住结果作为这个问题的答案（对于其他人）。

Following is (more or less) the original single-broker file which you can obtain here .以下是（或多或少）您可以在此处获得的原始single-broker文件。 I added a static.network (called verilabs ) and removed two unneeded services as well as removed the control-center paid service.我添加了一个 static.network （称为verilabs ）并删除了两个不需要的服务以及删除了control-center付费服务。

The A nswer to this Q uestion provides the three-broker version.该问题的答案提供了three-broker版本。

---
version: '3'

# Single broker version.

networks:
  verilabs:
    name: verilabs
    external: false
    driver: bridge
    ipam:
     config:
       - subnet: 192.168.10.0/24
         gateway: 192.168.10.1

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    hostname: zookeeper
    container_name: zookeeper
    privileged: true
    ports:
      - "2181:2181"
    networks:
      verilabs:
        ipv4_address: 192.168.10.20
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:6.2.0
    hostname: broker
    container_name: broker
    privileged: true
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"
    networks:
      verilabs:
        ipv4_address: 192.168.10.21
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

  schema-registry:
    image: confluentinc/cp-schema-registry:6.2.0
    hostname: schema-registry
    container_name: schema-registry
    privileged: true
    depends_on:
      - broker
    ports:
      - "8081:8081"
    networks:
      verilabs:
        ipv4_address: 192.168.10.22
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

  connect:
    image: cnfldemos/kafka-connect-datagen:0.5.0-6.2.0
    hostname: connect
    container_name: connect
    privileged: true
    depends_on:
      - broker
      - schema-registry
    ports:
      - "8083:8083"
    networks:
      verilabs:
        ipv4_address: 192.168.10.23
    environment:
      CONNECT_BOOTSTRAP_SERVERS: 'broker:29092'
      CONNECT_REST_ADVERTISED_HOST_NAME: connect
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR

  ksqldb-server:
    image: confluentinc/cp-ksqldb-server:6.2.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    privileged: true
    depends_on:
      - broker
      - connect
    ports:
      - "8088:8088"
    networks:
      verilabs:
        ipv4_address: 192.168.10.24
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_BOOTSTRAP_SERVERS: "broker:29092"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      KSQL_KSQL_CONNECT_URL: "http://connect:8083"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true'
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true'

  rest-proxy:
    image: confluentinc/cp-kafka-rest:6.2.0
    depends_on:
      - broker
      - schema-registry
    ports:
      - 8082:8082
    networks:
      verilabs:
        ipv4_address: 192.168.10.25
    hostname: rest-proxy
    container_name: rest-proxy
    privileged: true
    environment:
      KAFKA_REST_HOST_NAME: rest-proxy
      KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092'
      KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
      KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'

2 个解决方案

解决方案1
 0  2021-10-12 21:31:46

Here's the three-broker version.这是three-broker版本。 You can run sdiff(1) on both files to spot the differences and pattern;您可以在两个文件上运行sdiff(1)以发现差异和模式； then generalize to more brokers if needed.如果需要，然后推广到更多的经纪人。

PS : I added a KAFKA_OPTS to the Zookeeper service, which enables you to run four-letter-word commands to query it's status. PS ：我在Zookeeper服务中添加了一个KAFKA_OPTS ，它使您可以运行four-letter-word命令来查询它的状态。 (eg ruok , stat ,...). （例如ruok ， stat ，...）。 Without that whitelisting , those commands would be rejected.如果没有那个whitelisting ，这些命令将被拒绝。

---
version: '3'

# Three broker version.

networks:
  verilabs:
    name: verilabs
    external: false
    driver: bridge
    ipam:
     config:
       - subnet: 192.168.10.0/24
         gateway: 192.168.10.1

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    hostname: zookeeper
    container_name: zookeeper
    privileged: true
    ports:
      - "2181:2181"
    networks:
      verilabs:
        ipv4_address: 192.168.10.20
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      KAFKA_OPTS: "-Dzookeeper.4lw.commands.whitelist=*"

  broker01:
    image: confluentinc/cp-kafka:6.2.0
    hostname: broker01
    container_name: broker01
    privileged: true
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"
    networks:
      verilabs:
        ipv4_address: 192.168.10.21
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker01:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

  broker02:
    image: confluentinc/cp-kafka:6.2.0
    hostname: broker02
    container_name: broker02
    privileged: true
    depends_on:
      - zookeeper
    ports:
      - "29093:29092"
      - "9093:9092"
      - "9102:9101"
    networks:
      verilabs:
        ipv4_address: 192.168.10.22
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker02:29092,PLAINTEXT_HOST://localhost:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

  broker03:
    image: confluentinc/cp-kafka:6.2.0
    hostname: broker03
    container_name: broker03
    privileged: true
    depends_on:
      - zookeeper
    ports:
      - "29094:29092"
      - "9094:9092"
      - "9103:9101"
    networks:
      verilabs:
        ipv4_address: 192.168.10.23
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker03:29092,PLAINTEXT_HOST://localhost:9094
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

  schema-registry:
    image: confluentinc/cp-schema-registry:6.2.0
    hostname: schema-registry
    container_name: schema-registry
    privileged: true
    depends_on:
      - broker01
      - broker02
      - broker03
    ports:
      - "8081:8081"
    networks:
      verilabs:
        ipv4_address: 192.168.10.24
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker01:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

  connect:
    image: cnfldemos/kafka-connect-datagen:0.5.0-6.2.0
    hostname: connect
    container_name: connect
    privileged: true
    depends_on:
      - broker01
      - broker02
      - broker03
      - schema-registry
    ports:
      - "8083:8083"
    networks:
      verilabs:
        ipv4_address: 192.168.10.25
    environment:
      CONNECT_BOOTSTRAP_SERVERS: 'broker01:29092'
      CONNECT_REST_ADVERTISED_HOST_NAME: connect
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR

  ksqldb-server:
    image: confluentinc/cp-ksqldb-server:6.2.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    privileged: true
    depends_on:
      - broker01
      - broker02
      - broker03
      - connect
    ports:
      - "8088:8088"
    networks:
      verilabs:
        ipv4_address: 192.168.10.26
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_BOOTSTRAP_SERVERS: "broker01:29092"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      KSQL_KSQL_CONNECT_URL: "http://connect:8083"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true'
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true'

  rest-proxy:
    image: confluentinc/cp-kafka-rest:6.2.0
    depends_on:
      - broker01
      - broker02
      - broker03
      - schema-registry
    ports:
      - 8082:8082
    networks:
      verilabs:
        ipv4_address: 192.168.10.27
    hostname: rest-proxy
    container_name: rest-proxy
    privileged: true
    environment:
      KAFKA_REST_HOST_NAME: rest-proxy
      KAFKA_REST_BOOTSTRAP_SERVERS: 'broker01:29092'
      KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
      KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'

I hope this helps others.我希望这对其他人有帮助。

解决方案2
 0  2022-03-07 12:58:43

I believe the existing answer has some typos in the ports as the brokers would not connect to each other for me even though zookeeper saw them (two in my case).我相信现有的答案在端口中有一些拼写错误，因为即使 zookeeper 看到它们（在我的情况下是两个），经纪人也不会为我相互连接。

The following seems to have solved things for me.以下似乎已经为我解决了问题。

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker01:
    image: confluentinc/cp-server:6.2.1
    hostname: broker01
    container_name: broker01
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker01:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

  broker02:
    image: confluentinc/cp-server:6.2.1
    hostname: broker02
    container_name: broker02
    depends_on:
      - zookeeper
    ports:
      - "39092:39092"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker02:9092,PLAINTEXT_HOST://localhost:39092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 什么是 docker-compose.yml 文件？ - What is a docker-compose.yml file? 自定义文件而不是 docker-compose.yml - Custom file instead of docker-compose.yml 如何创建 docker-compose.yml 文件 - How To Create docker-compose.yml file 这个 docker-compose.yml 文件有什么问题 - What is wrong with this docker-compose.yml file docker-compose.yml 文件中的指令问题 - docker-compose.yml problem with directives in file docker-compose.yml文件中＆符号的含义 - Meaning of ampersand (&) in docker-compose.yml file 将 docker-compose.yml 文件转换为 kubernetes - convert docker-compose.yml file to kubernetes 在 docker-compose.yml 文件中创建目录 - Create a directory in docker-compose.yml file Docker：如何连接到 docker-compose.yml 文件中未定义的 kafka 容器 - Docker: How to connect to a kafka container which is not defined in docker-compose.yml file 如何为Kafka本地开发配置docker-compose.yml？ - How to configure docker-compose.yml for Kafka local development?
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM