Dockerized Spring Cloud Stream services with Kafka broker unable to connect to Zookeeper

Question

I'm testing a sample spring cloud stream application (running on a Ubuntu linux machine) with one source and one sink services. All my services are docker-containerized and I would like to use kafka as message broker.

Below the relevant parts of the docker-compose.yml :

  zookeeper:
    image: confluent/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"


  kafka:
    image: wurstmeister/kafka:0.9.0.0-1
    container_name: kafka
    ports:
      - "9092:9092"
    links:
      - zookeeper:zk
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=192.168.33.101
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_DELETE_TOPIC_ENABLE=true
      - KAFKA_LOG_RETENTION_HOURS=1
      - KAFKA_MESSAGE_MAX_BYTES=10000000
      - KAFKA_REPLICA_FETCH_MAX_BYTES=10000000
      - KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000
      - KAFKA_NUM_PARTITIONS=2
      - KAFKA_DELETE_RETENTION_MS=1000

    .
    .
    .

  # not shown: eureka service registry, spring cloud config service, etc.

  myapp-service-test-source:
    container_name: myapp-service-test-source
    image: myapp-h2020/myapp-service-test-source:0.0.1
    environment:
      SERVICE_REGISTRY_HOST: 192.168.33.101
      SERVICE_REGISTRY_PORT: 8761
    ports:
      - 8081:8080

    .
    .
    .

Here the relevant part of application.yml for my service-test-source service:

spring:
  cloud:
    stream:
      defaultBinder: kafka
      bindings:
        output:
          destination: messages
          content-type: application/json
      kafka:
        binder:
          brokers: ${SERVICE_REGISTRY_HOST:192.168.33.101}
          zkNodes: ${SERVICE_REGISTRY_HOST:192.168.33.101}
          defaultZkPort: 2181 
          defaultBrokerPort: 9092

The problem is the following, if I launch the docker-compose above, in the test-source container log I notice that the service fails to connect to zookeeper, giving a repeated set of Connection refused error, and finishing with a ZkTimeoutException which makes the service terminate (see below).

The strange fact is that, if instead of running my source (and sink) test services as docker containers I run them as jar files via maven mvn spring-boot:run <etc...> the services work fine and are able to exchange messages via kafka. (note that kafka, zookeeper, etc. are still running as docker containers).

.
.
.

*** THE FOLLOWING REPEATED n TIMES ***
2017-02-14 14:40:09.164  INFO 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-14 14:40:09.166  WARN 1 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_111]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_111]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]


.
.
.

java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
        at java.lang.Thread.run(Thread.java:745)
        Caused by: org.springframework.context.ApplicationContextException: Failed to start bean 'outputBindingLifecycle'; nested exception is org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000

Any idea what the problem might be?

edit:

I discovered that in the "jar" execution logs the test-source service tries to connect to zookeeper through the IP 127.0.0.1 , as can be seen from the log snipped below:

2017-02-15 14:24:04.159  INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:04.159  INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:04.178  INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-02-15 14:24:04.201  INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15a421fd9ec000a, negotiated timeout = 10000
2017-02-15 14:24:05.870  INFO 10348 --- [           main] org.apache.zookeeper.ZooKeeper           : Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@72ba68e3
2017-02-15 14:24:05.882  INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-02-15 14:24:05.883  INFO 10348 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session

This explains why everything works on the jar execution but not the docker one (the zookeeper container exports its 2181 port to the host machine, so it's visible as localhost for the service process when running directly on the host machine), but doesn't solve the problem: Apparently the spring cloud stream kafka configuration is ignoring the property spring.cloud.stream.kafka.binder.zkNodes as set in the application.yml (note that if I log the value of such environment variable from the service, I see the correct value of 192.168.33.101 that I hardcoded there for debugging purposes).

Answer 1

You have set the defaultBinder to be rabbit while trying to use the Kafka binder configuration. Do you have both rabbit and kafka binders in the classpath of your application? In that case, you can enable here

Answer 2

zookeeper:

image: wurstmeister/zookeeper

container_name: 'zookeeper'

ports:
  - 2181:2181

--------------------- kafka --------------------------------

kafka:

image:  wurstmeister/kafka
container_name: 'kafka'
environment:
  - KAFKA_ADVERTISED_HOST_NAME=kafka
  - KAFKA_ADVERTISED_PORT=9092
  - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
  - KAFKA_CREATE_TOPICS=kafka_docker_topic:1:1
ports:
  - 9092:9092
depends_on:
  - zookeeper

---

spring:

profiles: dev

cloud:

stream:

defaultBinder: kafka

kafka:

   binder:

      brokers: kafka     # i added brokers and zkNodes property

      zkNodes: zookeeper #

        bindings:

        input:

        destination: message

        content-type: application/json