在虚拟机中的docker中与kafka连接

Question

I am trying to use wurstmeister\\kafka-docker image with docker-compose, but I am having real problems with connecting everything.

I am working on Oracle Linux 7.6 as guest OS based on XEN host server so I don't have virtualization in this virtual machine.

I am trying to connect with one broker so I used docker-compose-single-broker.yml :

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

So when I run docker-compose -f docker-compose-single-broker.yml up everything looks good:

kafka_1      | creating topics: test:1:1
zookeeper_1  | 2019-04-23 12:32:16,680 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /172.24.0.2:50096
zookeeper_1  | 2019-04-23 12:32:16,685 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /172.24.0.2:50096
zookeeper_1  | 2019-04-23 12:32:16,690 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@694] - Established session 0x100476bb75e0001 with negotiated timeout 30000 for client /172.24.0.2:50096
zookeeper_1  | 2019-04-23 12:32:17,161 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x100476bb75e0001
zookeeper_1  | 2019-04-23 12:32:17,165 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - Closed socket connection for client /172.24.0.2:50096 which had sessionid 0x100476bb75e0001

After that I open new terminal and run ssh: docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e HOST_IP=172.17.0.1 -e ZK=172.17.0.1:2181 -i -t kafka-docker_kafka /bin/bash

But when I try to create new topic:

kafka-topics.sh --create --topic topicELO --partitions 4 --zookeeper $ZK --replication-factor 1

I've got exception:

Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
        at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:96)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
        at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

I also tried localhost and IP of my machine - same result.

When I try to do that in Docker for Windows, using host.docker.internal everything works fine - where I made a mistake on linux?

Answer 1

You need to add the --network option to be the same name as the Docker-Compose network on your docker run command.

Otherwise your "new container" won't be in the same network as the Zookeeper container, and therefore a connection will timeout.

Once you do that, you don't need -e ZK , as you would just use kafka-topics --zookeeper zookeeper:2181 (the container service name)

Use docker network ls and look for the one that contains the directory name you started docker-compose in

---

The quicker way would be to do this, though

docker-compose exec kafka bash -c "kafka-topics --create --zookeeper zookeeper:2181 ... "

---

Note: KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1 isn't "correct" as you would be telling any external Kafka clients to connect to themselves (on 127.0.0.1), not actually the Kafka container. more details here