LEADER_NOT_AVAILABLE - docker container

Question

setting up docker containers (kafka, zookeeper) and trying to (publish and) consume from another, receive the following error: KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"} There is no additional error indication when publishing or consuming. When testing connectivity with telnet 172.18.0.3 9092 I receive the expected result.

In [2]: c.list_topics()
Out[2]: ClusterMetadata(9ToJF8nPQC-rCTXGxuUalw)
In [3]: l = c.list_topics()
In [4]: l.brokers
Out[4]: {1010: BrokerMetadata(1010, 172.18.0.3:9092)}
In [7]: l.orig_broker_name
Out[7]: u'172.18.0.3:9092/1010'
In [8]: l.topics
Out[8]:
{'__consumer_offsets': TopicMetadata(__consumer_offsets, 50 partitions),
 'item': TopicMetadata(item, 1 partitions),
 'mytopic': TopicMetadata(mytopic, 1 partitions)}
In [9]: i  =l.topics['item']
In [10]: i.partitions
Out[10]: {0: PartitionMetadata(-1, KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"})}

docker-compose.yml:

version: '2'
services:
  zookeeper:
    restart: always
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    restart: always
    depends_on:
      - zookeeper
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "172.18.0.3"
      KAFKA_ADVERTISED_PORT: "9092"
      KAFKA_CREATE_TOPICS: "item:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

ubuntu@STAGE:~/docker/kafka# sudo docker network inspect kafka_default

[
    {
        "Name": "kafka_default",
        "Id": "fc8afcde4318599b66f540ebd6221cffe57382a70b43ed1d58eda7fe059e7a18",
        "Created": "2018-07-25T13:29:56.034567548Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "7776db25d1ca9a3ec98775ff0dabe356409dda9f782a771c9b1f7ae4e04f3453": {
                "Name": "test",
                "EndpointID": "204158f6f167b12c06b2dcdda856992c5ea75adc80b06ac00f9abebb4f0179ba",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            },
            "b4869e44cf136a129701413c5a5dd371d15d6158bfaf577d30aeb6ad66b63263": {
                "Name": "kafka_zookeeper_1", DEPTH_PRIORITY = 1
                "EndpointID": "229b2587d92c8593d933d2583e09e669aa6f9a9c04e5b94ebf5a25be57aad27f",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "f682fd27ef52a40abf501348b5b16e59b48440041d1e4827ddb15a0ef8dbdd14": {
                "Name": "kafka_kafka_1",
                "EndpointID": "b17874d64c699635c471a1bcb515cb1441d22b2896e98a113115a4239caed460",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    } ]

Answer 1

Always try to use public ip in the KAFKA_ADVERTISED_HOST_NAME, and you can use KAFKA_HOST_NAME to specify the hostname of your instance.

Also if you are using the recent kafka version, use listeners and advertised.listeners instead of using host.name and advertised.host.name.