没有docker-compose的Docker上的Kafka？

Question

I wanted to set up kafka single broker on my local machine(Windows 7) using Docker. Essentially, I used the steps listed out on this link:

Apache Kafka on Docker

Then, I was able to create a topic by running the following command:

kafka-topics.bat --create --zookeeper 192.168.59.103:32769 --replica 1 --partition 1 --topic test 

This created the topic alright as when I do list-topics, it displays correctly.

Next, I tried to publish a message using the kafka-console-producer.bat file and when I do that and hit enter after entering the message, I get the following exception:

[2015-08-02 13:48:46,075] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,084] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,085] ERROR Failed to collate messages by topic, partition d
ue to: Failed to fetch topic metadata for topic: test (kafka.producer.async.Defa
ultEventHandler)
[2015-08-02 13:48:46,195] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,205] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,206] ERROR Failed to collate messages by topic, partition d
ue to: Failed to fetch topic metadata for topic: test (kafka.producer.async.Defa
ultEventHandler)
[2015-08-02 13:48:46,315] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,325] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,325] ERROR Failed to collate messages by topic, partition d
ue to: Failed to fetch topic metadata for topic: test (kafka.producer.async.Defa
ultEventHandler)
[2015-08-02 13:48:46,434] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,442] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,443] ERROR Failed to collate messages by topic, partition d
ue to: Failed to fetch topic metadata for topic: test (kafka.producer.async.Defa
ultEventHandler)
[2015-08-02 13:48:46,548] WARN Error while fetching metadata [{TopicMetadata for
 topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableExcep
tion}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.
producer.BrokerPartitionInfo)
[2015-08-02 13:48:46,549] ERROR Failed to send requests for topics test with cor
relation ids in [9,17] (kafka.producer.async.DefaultEventHandler)
[2015-08-02 13:48:46,549] ERROR Error in handling batch of 1 events (kafka.produ
cer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries
.
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.s
cala:90)
        at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThrea
d.scala:105)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.appl
y(ProducerSendThread.scala:88)
        at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.appl
y(ProducerSendThread.scala:68)
        at scala.collection.immutable.Stream.foreach(Stream.scala:547)
        at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThr
ead.scala:67)
        at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:
45)

The exception seems strange to me as when I started the kafka using

docker run --name kafka -e HOST_IP=localhost -e KAFKA_ADVERTISED_HOST_NAME=192.168.59.103 -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_BROKER_ID=1 -e ZK=zk -p 9092 --link zookeeper:zk -t wurstmeister/kafka

The last line that shows up in the terminal is :

[2015-08-02 17:20:44,037] INFO Registered broker 1 at path /brokers/ids/1 with a
ddress 192.168.59.103:9092. (kafka.utils.ZkUtils$)
[2015-08-02 17:20:44,055] INFO [Kafka Server 1], started (kafka.server.KafkaServ
er)
[2015-08-02 17:20:44,108] INFO New leader is 1 (kafka.server.ZookeeperLeaderElec
tor$LeaderChangeListener)

So, I am guessing that the leader has already been assigned. And lastly, from the Consumer side, when I run the bat file, the following is thrown:

[2015-08-02 13:41:47,005] WARN Fetching topic metadata with correlation id 0 for
 topics [Set(test)] from broker [id:1,host:192.168.59.103,port:9092] failed (kaf
ka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
        at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncP
roducer.scala:72)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
        at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(Consu
merFetcherManager.scala:66)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

So, what is it that I am doing wrongly over here? Do I always need compose for running kafka? My understanding is that compose is needed if I want to run multiple brokers which isn't what I am doing anyways. Wouldn't linking the 2 containers suffice in my case?

Answer 1

I noticed that the ZooKeeper host & port that you passed to kafka-topcs.bat is not the default one. Perhaps that is the source of the problem. Did you start a separate Docker container for Zookeeper?