[英]Connecting Kafka-Python with a cluster with Kerberos
I'm trying to connect to a Kafka with Kafka-Python, the Kafka cluster has Kerberos that we need to build some commands to do few steps. 我正在尝试使用Kafka-Python连接到Kafka,Kafka集群具有Kerberos,我们需要构建一些命令来执行几个步骤。
I have created one Topic at the cluster and I did the test with ./kafka-console-producer.sh
and ./kafka-console-consumer.sh
and works really well. 我在集群中创建了一个主题,我使用
./kafka-console-producer.sh
和./kafka-console-consumer.sh
进行了测试,效果非常好。
But when I try to connect with Kafka-Python I had a problem. 但是当我尝试连接Kafka-Python时,我遇到了问题。 See my code below:
请参阅下面的代码:
def produce():
print ('Producer')
k_producer = KafkaProducer(bootstrap_servers='hostname:6667',
security_protocol='SASL_PLAINTEXT',
sasl_plain_username='machine_usr',
sasl_plain_password='machine_pwd',
sasl_mechanism='PLAIN')
for i in range(10):
print ('Before send')
k_producer.send('myTopic', 'Testing My Topic ' + str(i))
print ('After send')
Well, running this stuff I got this erro message after 30 secconds: 好吧,运行这些东西我在30秒后得到了这个错误信息:
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\producer\kafka.py", line 328, in __init__
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 202, in __init__
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 791, in check_version
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
I'm running it in a remote machine. 我在远程机器上运行它。 And the bootstrap_server I used the Zookeeper hostname and port but didn't work as well.
而bootstrap_server我使用了Zookeeper主机名和端口,但也没有用。
I found few things about it, and in the git of the Kafka-Python I found that they had implemented . 我发现了一些关于它的东西,并且在Kafka-Python的git中我发现它们已经实现了 。
Is there other way to connect? 还有其他连接方式吗?
Well Guys, 好伙计们,
I found the issue. 我发现了这个问题。
The problem is that Kerberos is not supported for Kafka producer in Python using Key Tab. 问题是使用Key Tab,Python中的Kafka生产者不支持Kerberos。
To use Key Tab we need to set a java Environment Variable. 要使用Key Tab,我们需要设置一个java Environment Variable。
According Hortonworks we need to set the client_jaas_client to connect. 根据Hortonworks,我们需要将client_jaas_client设置为connect。
The solution was using Py4j to call the Kafka Producer in JVM. 解决方案是使用Py4j在JVM中调用Kafka Producer。
If the task is to solve this problem in python, another alternative could be to use confluent-kafka-python library that internally uses librdkafka that is written in C, and supports SASL, and the use of the keytab file. 如果任务是在python中解决这个问题,另一种选择可能是使用confluent-kafka-python库,它在内部使用用C编写的librdkafka ,并支持SASL,以及keytab文件的使用。 That wouldn't require having a separate Java process for the communication with kafka over SASL.
这不需要单独的Java进程来通过SASL与kafka进行通信。
For instructions also refer to the documentation of the librdkafka library: 有关说明,请参阅librdkafka库的文档:
https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka - general intro https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md - the properties that can be passed into the constructor of confluent_kafka.Producer
and confluent_kafka.Consumer
in python https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka - 一般介绍https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md - 可以传递的属性到python中的
confluent_kafka.Producer
和confluent_kafka.Consumer
的构造函数
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.