Error with docker-compose up when running a micro service with kafka and zookeeper

Question

SOLVED! thank you guys, i am going to update the correct config to work.

I'm trying to do my first microservice with spring-boot, kafka and zookeeper through docker-compose. I'm following in the footsteps of https://docs.docker.com/compose/gettingstarted/ but when I try to raise the services with the docker-compose up command I'm getting an exception:

demo-quartz_1  | web - 2018-05-09 14:57:48,824 [scheduler_Worker-2] INFO  c.a.d.service.SampleJobService - created entity...
demo-quartz_1  | web - 2018-05-09 14:57:48,824 [scheduler_Worker-2] INFO  c.a.d.service.SampleJobService - BitcoinEuroEntity [idBCEntity=8, id=bitcoin, name=Bitcoin, symbol=BTC, rank=1, priceUsd=9299.31, priceBtc=1.0, _24hVolumeUsd=7170100000.0, marketCapUsd=158311453440, availableSupply=17024000.0, totalSupply=17024000.0, maxSupply=21000000.0, percentChange1h=0.05, percentChange24h=1.36, percentChange7d=1.7, lastUpdated=1525877672, priceEur=7830.80946135, _24hVolumeEur=6037833658.5, marketCapEur=133311700270]
demo-quartz_1  | web - 2018-05-09 14:58:38,868 [scheduler_Worker-1] ERROR o.s.k.s.LoggingProducerListener - Exception thrown when sending a message with key='null' and payload='BitcoinEuroKafkaEntity [id=bitcoin, name=Bitcoin, symbol=BTC, rank=1, priceUsd=9299.31, priceBtc=1.0...' to topic aironman:
demo-quartz_1  | org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

I can see that zookeeper, kafka and micro service containers are running fine within their containers:

$:demo-quartz aironman$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                                NAMES
f9201046956c        wurstmeister/kafka                    "start-kafka.sh"         7 minutes ago       Up 6 minutes        0.0.0.0:9092->9092/tcp                               demo-quartz_kafka_1
386270f126da        aironman/demo-quartz:0.0.1-SNAPSHOT   "/usr/bin/java -jar …"   7 minutes ago       Up 6 minutes                                                             demo-quartz_demo-quartz_1
ac1df2c6429d        wurstmeister/zookeeper                "/bin/sh -c '/usr/sb…"   7 minutes ago       Up 6 minutes        22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   demo-quartz_zookeeper_1

This is the GitHub repo develop branch

Docker file looks like:

FROM openjdk:8-jre
ENTRYPOINT ["/usr/bin/java", "-jar", "/usr/share/aironman/demo-quartz.jar"]
ARG JAR_FILE
ADD target/${JAR_FILE} /usr/share/aironman/demo-quartz.jar

Docker-compose.yml file looks like:

version: '3.6'
services:
demo-quartz:
image: aironman/demo-quartz:0.0.1-SNAPSHOT
deploy:
  replicas: 5
  resources:
    limits:
      cpus: "0.5"
      memory: 512M
  restart_policy:
      condition: on-failure

  zookeeper:
  image: wurstmeister/zookeeper
  ports:
  - "2181:2181"

  kafka:
  image: wurstmeister/kafka
  ports:
  - "9092:9092"
  environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
  KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
  volumes:
  - /var/run/docker.sock:/var/run/docker.sock

This is a link of the complete output of docker-compose up command:

application.properties file of the micro service looks like:

using.spring.schedulerFactory=true
kafka.bootstrapAddress=kafka:9092
message.topic.name=aironman
greeting.topic.name=greeting
filtered.topic.name=filtered
partitioned.topic.name=partitioned

I have been looking at other threads in stackoverflow with similar problems but none of them help me to solve the problem by raising the microservice along with zookeeper and kafka talking to each other without any problem.

It looks like the micro service is not able to communicate with kafka and zookeeper isn't, because, docker-compose runs the container but micro service container cannot talk each other. If i run the process locally using mvn spring-boot:run and running locally zookeeper and kafka, it works fine, i can push the message into the topic.

Can you guide me to the right solution?

Thank you guys, i learnt a lot!

:demo-quartz aironman$ docker exec -it 693ab89a26fb  /bin/sh
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:13:00:03  
      inet addr:172.19.0.3  Bcast:172.19.255.255  Mask:255.255.0.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:227 errors:0 dropped:0 overruns:0 frame:0
      TX packets:390 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0 
      RX bytes:20677 (20.1 KiB)  TX bytes:30809 (30.0 KiB)

lo        Link encap:Local Loopback  
      inet addr:127.0.0.1  Mask:255.0.0.0
      UP LOOPBACK RUNNING  MTU:65536  Metric:1
      RX packets:6 errors:0 dropped:0 overruns:0 frame:0
      TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1 
      RX bytes:590 (590.0 B)  TX bytes:590 (590.0 B)

/ # ping 172.19.0.2
PING 172.19.0.2 (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.190 ms
64 bytes from 172.19.0.2: seq=1 ttl=64 time=0.167 ms
64 bytes from 172.19.0.2: seq=2 ttl=64 time=0.121 ms
64 bytes from 172.19.0.2: seq=3 ttl=64 time=0.173 ms
64 bytes from 172.19.0.2: seq=4 ttl=64 time=0.173 ms
^C
--- 172.19.0.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.121/0.164/0.190 ms
/ # 
/ # 
/ # 
/ # kafka-topics.sh --list --zookeeper 172.19.0.2:2181
/ # kafka-topics.sh --create --zookeeper 172.19.0.2:2181 --replication-factor 1 
--partitions 1 --topic aironman 
Created topic "aironman".
/ # kafka-topics.sh --list --zookeeper 172.19.0.2:2181
aironman

I can see that the topic is created in kafka broker, but exception still happen.

Answer 1

I got it to work with this configuration for Kafka.

kafka:
image: wurstmeister/kafka
depends_on:
  - zookeeper
ports:
  - "9092:9092"
environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
  KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092

The key configuration is to have the docker service name and the advertised host name the same (kafka in this case).In your application.properties, you can use kafka.bootstrapAddress=kafka:9092.

Since all the docker services are written in the same docker-compose.yml, all of them will participate in the same default network and will be able to resolve each other by the service name.

Update : Using docker-compose from the original question and with few updates.

 version: '3.3'
    services:
      demo-quartz:
        image: aironman/demo-quartz:0.0.1-SNAPSHOT

      zookeeper:
        image: wurstmeister/zookeeper
        ports:
          - "2181:2181"
      kafka:
        image: wurstmeister/kafka
        ports:
          - "9092:9092"
        environment:
          KAFKA_BROKER_ID: 1
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
          KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092

This works perfectly fine. Have you updated the bootstrap url to bootstrap.servers = kafka:9092? Also, demo-quartz is using old version of Kafka Client. Please update.

web - 2018-05-09 18:17:29,260 [scheduler_Worker-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version : 0.10.1.1
web - 2018-05-09 18:17:29,260 [scheduler_Worker-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId : f10ef2720b03b247

You can build a sample kafka producer docker image from here: https://github.com/vallikranth/kafka-spring-producer and verify your setup.

Answer 2

I am by no means a Docker expert but I believe that the default network type that is created is a bridge network. With bridge networks you need to link services explicitly so that they can see each other.

For example: version: '3.6' services: demo-quartz: image: aironman/demo-quartz:0.0.1-SNAPSHOT deploy: replicas: 5 resources: limits: cpus: "0.5" memory: 512M restart_policy: condition: on-failure links: - kafka - zookeeper zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka ports: - "9092:9092" links: - zookeeper environment: KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 volumes: - /var/run/docker.sock:/var/run/docker.sock