Does Spring Cloud Dataflow Local Server need to connect zookeeper?

Question

I am trying to set up scdf on docker containers.

All the containers are in one network

The two kafka s are exposed by port 2181

and I am sure the kafka s are connecting to the zookeeper s. (if not, they will not be started normally)

But during the creation of the simple stream

stream create --name test --definition "time | log" --deploy

I got these logs in the stdout_0.log file which stands for the log app logs.

2016-12-14 01:18:03.264  INFO 62 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-12-14 01:18:03.265  WARN 62 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_111-internal]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_111-internal]
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]

in stderr_1.log , I have

Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
    at org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.ensureTopicCreated(KafkaMessageChannelBinder.java:354)
    at org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.createKafkaConsumer(KafkaMessageChannelBinder.java:441)
    at org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.doBindConsumer(KafkaMessageChannelBinder.java:282)
    at org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.doBindConsumer(KafkaMessageChannelBinder.java:121)
    at org.springframework.cloud.stream.binder.AbstractBinder.bindConsumer(AbstractBinder.java:177)
    at org.springframework.cloud.stream.binding.ChannelBindingService.bindConsumer(ChannelBindingService.java:93)
    at org.springframework.cloud.stream.binding.BindableProxyFactory.bindInputs(BindableProxyFactory.java:189)
    at org.springframework.cloud.stream.binding.InputBindingLifecycle.start(InputBindingLifecycle.java:55)
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
    ... 27 more

The first log said I am using localhost:0.0.0.0:2181 for zookeeper connection, which I am not, and I have configured the real IP address for kafka .

the second log said the timeout for connection is 10000 , which in kafka, I set the connection timeout to 6000

So these are not from kafka , then they are from scdf .

the fact is, when I look up the reference of scdf link , here it says, the zookeeper is not used ( or maybe not directly used ) in this architecture.

which confused me... if these logs not from kafka and not from scdf , then where does them from?

If scdf does use zookeeper , then how to configure the connection for zookeeper ?

UPDATE

I am thinking about when scdf is trying to create a topic , from what I know about creating a topic you have to specify the zookeeper connection, scdf uses localhost for producer s or consumer s to register on zookeeper , so scdf is actually using zookeeper somehow. but how to configure this?

Answer 1

OK, after a dig through the reference file. I got this:

--spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers=192.168.1.100:9092
--spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes=192.168.1.100:2181

Hope this will help the ones who are looking for the answer....