[英]Unable to produce/consume to Kafka on Kubernetes when using loadBalancer Service
Background背景
I am running Kafka on kuberentes using confluent open source helm charts.我在 kuberentes 上使用融合的开源 helm 图表运行 Kafka。 I already have an eks cluster running with managed node groups.
我已经有一个运行托管节点组的 eks 集群。
When i expose the brokers using NodePort it works fine.当我使用 NodePort 公开代理时,它工作正常。 However i want to enable load balancer, I am able to enable it and service is created per broker pod.
但是我想启用负载均衡器,我能够启用它并且每个代理 pod 创建服务。 (Enabled an internal Network load balancer) All our producers are in aws.
(启用内部网络负载均衡器)我们所有的生产者都在 aws 中。
$ kubectl get svc -n kafka
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kafka-0-external LoadBalancer 10.100.185.40 ac5b7fccb69bc4738b2e498995e65de2-9d6b81206f5d1d7d.elb.us-east-2.amazonaws.com 31090:30366/TCP 10m
kafka-1-external LoadBalancer 10.100.192.249 ae035d93de7874c49bc2402d5c174403-65cdb5cda161fa89.elb.us-east-2.amazonaws.com 31090:31063/TCP 10m
kafka-2-external LoadBalancer 10.100.80.80 a36dc44c757f4429b81163ab651a7012-e94e40584210b988.elb.us-east-2.amazonaws.com 31090:32700/TCP 10m
kafka-cp-kafka ClusterIP 10.100.163.158 <none> 9092/TCP 10m
kafka-cp-kafka-connect ClusterIP 10.100.139.66 <none> 8083/TCP 10m
kafka-cp-kafka-headless ClusterIP None <none> 9092/TCP 10m
kafka-cp-kafka-rest ClusterIP 10.100.146.106 <none> 8082/TCP 10m
kafka-cp-schema-registry ClusterIP 10.100.103.114 <none> 8081/TCP 10m
kafka-cp-zookeeper NodePort 10.100.22.195 <none> 2181:32724/TCP 10m
kafka-cp-zookeeper-headless ClusterIP None <none> 2888/TCP,3888/TCP 10m
Now i want to test by producing and consuming.现在我想通过生产和消费来测试。 I started a new ec2 instance in the same vpc.
我在同一个 vpc 中启动了一个新的 ec2 实例。 I can get metadata but i cannot produce and consume.
我可以获得元数据,但我不能生产和消费。
ubuntu@ip-192-168-87-196:~/kafka_2.11-2.3.1/bin$ kafkacat -b ae035d93de7874c49bc2402d5c174403-65cdb5cda161fa89.elb.us-east-2.amazonaws.com:31090 -L
Metadata for all topics (from broker -1: ae035d93de7874c49bc2402d5c174403-65cdb5cda161fa89.elb.us-east-2.amazonaws.com:31090/bootstrap):
3 brokers:
broker 0 at kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090
broker 2 at kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090
broker 1 at kafka-cp-kafka-1.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090
8 topics:
topic "test" with 25 partitions:
partition 0, leader 1, replicas: 1,2,0, isrs: 1,0,2
partition 5, leader 0, replicas: 0,2,1, isrs: 1,0,2
partition 10, leader 2, replicas: 2,1,0, isrs: 1,0,2
When i try to produce i get this error当我尝试生产时出现此错误
ubuntu@ip-192-168-87-196:~/kafka_2.11-2.3.1/bin$ kafkacat -b ae035d93de7874c49bc2402d5c174403-65cdb5cda161fa89.elb.us-east-2.amazonaws.com:31090 -C -t test
% ERROR: Local: Host resolution failure: kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090/0: Failed to resolve 'kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090': Temporary failure in name resolution
% ERROR: Local: Host resolution failure: kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090/2: Failed to resolve 'kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090': Temporary failure in name resolution
% ERROR: Local: Host resolution failure: kafka-cp-kafka-1.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090/1: Failed to resolve 'kafka-cp-kafka-1.kafka-cp-kafka-headless.kafka.svc.cluster.local:31090': Temporary failure in name resolution
These are my listeners这些是我的听众
$ kubectl logs kafka-cp-kafka-2 -n kafka -c cp-kafka-broker | grep -i listeners
+ export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
advertised.listeners = PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
listeners = PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:31090
advertised.listeners = PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
listeners = PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:31090
advertised.listeners = PLAINTEXT://kafka-cp-kafka-2.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://:31090
listeners = PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:31090
I tried for a few days now and want some guidance.我现在尝试了几天,希望得到一些指导。 Let me know if anyone has anything to share, what am i missing?
如果有人有什么要分享的,请告诉我,我错过了什么?
If you are running kafka client outside the k8s cluster you have to use external IP or hostname visible outside of the cluster for KAFKA_ADVERTISED_LISTENERS:如果您在 k8s 集群外运行 kafka 客户端,则必须使用外部 IP 或 KAFKA_ADVERTISED_LISTENERS 在集群外可见的主机名:
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://ac5b7fccb69bc4738b2e498995e65de2-9d6b81206f5d1d7d.elb.us-east-2.amazonaws.com:30366 KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-cp-kafka-0.kafka-cp-kafka-headless.kafka:9092,EXTERNAL://ac5b7fccb69bc4738b2e498995e65de2-9d6b81206f5d1d7d.elb.us-east-2.amazonaws.com:30366
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.