Kafka Kubernetes：活动代理的数量“0”不满足偏移主题所需的复制因子“1”

Question

I'm trying to set up a Kafka pod in Kubernetes but I keep getting this error:

[2020-08-30 11:23:39,354] ERROR [KafkaApi-0] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

This is my Kafka deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka
  namespace: instagnam
  labels:
    app: instagnam
    service: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: instagnam
  template:
    metadata:
      labels:
        app: instagnam
        service: kafka
        id: "0"
    spec:
      containers:
       - name: kafka
         image: wurstmeister/kafka
         imagePullPolicy: Always
         ports:
         - containerPort: 9092
           name: kafka
         env:
         - name: KAFKA_PORT
           value: "9092"
         - name: KAFKA_ADVERTISED_PORT
           value: "9092"
         - name: KAFKA_ADVERTISED_HOST_NAME
           value: kafka
         - name: KAFKA_ZOOKEEPER_CONNECT
           value: zookeeper:2181
         - name: KAFKA_CREATE_TOPICS
           value: connessioni:2:1,ricette:2:1
         - name: KAFKA_BROKER_ID
           value: "0"

This is my Kafka service:

apiVersion: v1
kind: Service
metadata:
  name: kafka
  namespace: instagnam
  labels:
    app: instagnam
    service: kafka
spec:
  selector:
    app: instagnam
    service: kafka
    id: "0"
  type: LoadBalancer
  ports:
  - name: kafka
    protocol: TCP
    port: 9092

This is my Zookeeper deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper
  namespace: instagnam
  labels:
    app: instagnam
    service: zookeeper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: instagnam
      service: zookeeper
  template:
    metadata:
      labels:
        app: instagnam
        service: zookeeper
    spec:
      containers:
      - image: wurstmeister/zookeeper
        name: zookeeper
        imagePullPolicy: Always
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zookeeper

And this is my Zookeeper service: apiVersion: v1

kind: Service
metadata:
  name: zookeeper
  namespace: instagnam
spec:
  selector:
    app: instagnam
    service: zookeeper
  ports:
  - name: client
    protocol: TCP
    port: 2181
  - name: follower
    protocol: TCP
    port: 2888
  - name: leader
    protocol: TCP
    port: 3888

What am I doing wrong here?

If you need the full Kafka log here it is: https://pastebin.com/eBu8JB8A

And there are the Zookeper logs if you need them too: https://pastebin.com/gtnxSftW

EDIT: I'm running this on minikube if this can help.

Answer 1

Kafka broker.id changes maybe cause this problem. Clean up the kafka metadata under zk, deleteall /brokers... note: kafka data will be lost

Answer 2

Assuming that you're on the same Kafka image. The solution that fixed the issue for me was:

Replacing the deprecated settings of KAFKA_ADVERTISED_PORT and KAFKA_ADVERTISED_HOST_NAME as detailed in the docker image README see current docs (or README commit pinned ). With KAFKA_ADVERTISED_LISTENERS and KAFKA_LISTENERS on which I had to add an "inside" and "outside" configurations.

Summarized from https://github.com/wurstmeister/kafka-docker/issues/218#issuecomment-362327563