Unable to produce/consume to Kafka on Kubernetes when using loadBalancer Service

Question

Background

I am running Kafka on kuberentes using confluent open source helm charts. I already have an eks cluster running with managed node groups.

When i expose the brokers using NodePort it works fine. However i want to enable load balancer, I am able to enable it and service is created per broker pod. (Enabled an internal Network load balancer) All our producers are in aws.

$ kubectl get svc -n kafka
NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)             AGE
kafka-0-external              LoadBalancer   10.100.185.40    ac5b7fccb69bc4738b2e498995e65de2-9d6b81206f5d1d7d.elb.us-east-2.amazonaws.com   31090:30366/TCP     10m
kafka-1-external              LoadBalancer   10.100.192.249   ae035d93de7874c49bc2402d5c174403-65cdb5cda161fa89.elb.us-east-2.amazonaws.com   31090:31063/TCP     10m
kafka-2-external              LoadBalancer   10.100.80.80     a36dc44c757f4429b81163ab651a7012-e94e40584210b988.elb.us-east-2.amazonaws.com   31090:32700/TCP     10m
kafka-cp-kafka                ClusterIP      10.100.163.158   <none>                                                                          9092/TCP            10m
kafka-cp-kafka-connect        ClusterIP      10.100.139.66    <none>                                                                          8083/TCP            10m
kafka-cp-kafka-headless       ClusterIP      None             <none>                                                                          9092/TCP            10m
kafka-cp-kafka-rest           ClusterIP      10.100.146.106   <none>                                                                          8082/TCP            10m
kafka-cp-schema-registry      ClusterIP      10.100.103.114   <none>                                                                          8081/TCP            10m
kafka-cp-zookeeper            NodePort       10.100.22.195    <none>                                                                          2181:32724/TCP      10m
kafka-cp-zookeeper-headless   ClusterIP      None             <none>                                                                          2888/TCP,3888/TCP   10m

Now i want to test by producing and consuming. I started a new ec2 instance in the same vpc. I can get metadata but i cannot produce and consume.

ubuntu@ip-192-168-87-196:~/kafka_2.11-2.3.1/bin$ kafkacat -b ae035d93de7874c49bc2402d5c174403-65cdb5cda161fa89.elb.us-east-2.amazonaws.com:31090 -L
Metadata for all topics (from broker -1: ae035d93de7874c49bc2402d5c174403-65cdb5cda161fa89.elb.us-east-2.amazonaws.com:31090/bootstrap):
 3 brokers:
  broker 0 at kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090
  broker 2 at kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090
  broker 1 at kafka-cp-kafka-1.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090
 8 topics:
  topic "test" with 25 partitions:
    partition 0, leader 1, replicas: 1,2,0, isrs: 1,0,2
    partition 5, leader 0, replicas: 0,2,1, isrs: 1,0,2
    partition 10, leader 2, replicas: 2,1,0, isrs: 1,0,2

When i try to produce i get this error

ubuntu@ip-192-168-87-196:~/kafka_2.11-2.3.1/bin$ kafkacat -b ae035d93de7874c49bc2402d5c174403-65cdb5cda161fa89.elb.us-east-2.amazonaws.com:31090 -C -t test
% ERROR: Local: Host resolution failure: kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090/0: Failed to resolve 'kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090': Temporary failure in name resolution
% ERROR: Local: Host resolution failure: kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090/2: Failed to resolve 'kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090': Temporary failure in name resolution
% ERROR: Local: Host resolution failure: kafka-cp-kafka-1.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090/1: Failed to resolve 'kafka-cp-kafka-1.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090': Temporary failure in name resolution

These are my listeners

$ kubectl logs kafka-cp-kafka-2 -n kafka -c cp-kafka-broker | grep -i listeners
+ export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
        advertised.listeners = PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
        listeners = PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:31090
        advertised.listeners = PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
        listeners = PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:31090
        advertised.listeners = PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
        listeners = PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:31090

I tried for a few days now and want some guidance. Let me know if anyone has anything to share, what am i missing?

Answer 1

If you are running kafka client outside the k8s cluster you have to use external IP or hostname visible outside of the cluster for KAFKA_ADVERTISED_LISTENERS:

KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://ac5b7fccb69bc4738b2e498995e65de2-9d6b81206f5d1d7d.elb.us-east-2.amazonaws.com:30366