Kubernetes - Controller 0's connection to broker was unsuccessful

Question

I'm trying to set up Kafka on Kubernetes, but I keep getting the error mentioned in the title.

The full error is:

[2022-06-29 12:17:11,732] WARN [RequestSendThread controllerId=0] 
Controller 0's connection to broker kafka-service:29092 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread) java.net.SocketTimeoutException: Failed to connect within 30000 ms
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:293)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:246)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

Pod Definition:

kind: Pod
apiVersion: v1
metadata:
  name: kafka
  labels:
    app: kafka
spec:
  containers:
  - name: kafka
    image: confluentinc/cp-kafka:6.2.0
    ports:
    - containerPort: 9092
    env:
    - name: KAFKA_ZOOKEEPER_CONNECT
      value: zoo1:2181
    - name: KAFKA_BROKER_ID
      value: "0"
    - name: KAFKA_LISTENERS
      value: "LISTENER_INTERNAL://0.0.0.0:29092,LISTENER_EXTERNAL://0.0.0.0:9092"
    - name: KAFKA_ADVERTISED_LISTENERS
      value: "LISTENER_INTERNAL://kafka-service:29092,LISTENER_EXTERNAL://kafka-service:9092"
    - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
      value: "LISTENER_EXTERNAL:PLAINTEXT,LISTENER_INTERNAL:PLAINTEXT"
    - name: KAFKA_INTER_BROKER_LISTENER_NAME
      value: "LISTENER_INTERNAL"
    - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
      value: "1"
    - name: KAFKA_MAX_REQUEST_SIZE
      value: "10485760"
    - name: KAFKA_MAX_MESSAGE_BYTES
      value: "10485760"
    - name: KAFKA_MAX_PARTITION_FETCH_BYTES
      value: "10485760"

Service - I've also tried using ClusterIP but the error stays:

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
spec:
  type:  NodePort
  selector:
    app: kafka
  ports:
    - name: kafka1
      port: 9092
      targetPort: 9092
    - name: kafka2
      port: 29092
      targetPort: 29092

Any idea what i might be doing wrong?

Answer 1

Given that your advertised listeners are the exact same service name, then you do not need both of them on different ports. Instead, you could have "internal" within the pod, and "external" across namespaces

eg when using default namespace.

- name: KAFKA_ADVERTISED_LISTENERS
  value: "LISTENER_INTERNAL://kafka-service:29092,LISTENER_EXTERNAL://kafka-service.default.svc.cluster.local:9092"

Also will want to add both ports

ports:
  - containerPort: 9092
  - containerPort: 29092

Unclear what your error is being logged from, but seems like you should be using an address closer to kafka-service.svc.cluster.local:9092 - ref Kubernetes DNS docs

---

Alternatively, https://strimzi.io has very in-depth documentation about running and using Kafka on Kubernetes.