简体   繁体   English

将Kafka-Python与具有Kerberos的集群连接

[英]Connecting Kafka-Python with a cluster with Kerberos

I'm trying to connect to a Kafka with Kafka-Python, the Kafka cluster has Kerberos that we need to build some commands to do few steps. 我正在尝试使用Kafka-Python连接到Kafka,Kafka集群具有Kerberos,我们需要构建一些命令来执行几个步骤。

I have created one Topic at the cluster and I did the test with ./kafka-console-producer.sh and ./kafka-console-consumer.sh and works really well. 我在集群中创建了一个主题,我使用./kafka-console-producer.sh./kafka-console-consumer.sh进行了测试,效果非常好。

But when I try to connect with Kafka-Python I had a problem. 但是当我尝试连接Kafka-Python时,我遇到了问题。 See my code below: 请参阅下面的代码:

def produce():
    print ('Producer')
    k_producer = KafkaProducer(bootstrap_servers='hostname:6667', 
                               security_protocol='SASL_PLAINTEXT',
                               sasl_plain_username='machine_usr',
                               sasl_plain_password='machine_pwd',
                               sasl_mechanism='PLAIN')
    for i in range(10):
        print ('Before send')
        k_producer.send('myTopic', 'Testing My Topic  ' + str(i))
        print ('After send')

Well, running this stuff I got this erro message after 30 secconds: 好吧,运行这些东西我在30秒后得到了这个错误信息:

File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\producer\kafka.py", line 328, in __init__
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 202, in __init__
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 791, in check_version
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

I'm running it in a remote machine. 我在远程机器上运行它。 And the bootstrap_server I used the Zookeeper hostname and port but didn't work as well. 而bootstrap_server我使用了Zookeeper主机名和端口,但也没有用。

I found few things about it, and in the git of the Kafka-Python I found that they had implemented . 我发现了一些关于它的东西,并且在Kafka-Python的git中我发现它们已经实现了

Is there other way to connect? 还有其他连接方式吗?

Well Guys, 好伙计们,

I found the issue. 我发现了这个问题。

The problem is that Kerberos is not supported for Kafka producer in Python using Key Tab. 问题是使用Key Tab,Python中的Kafka生产者不支持Kerberos。

To use Key Tab we need to set a java Environment Variable. 要使用Key Tab,我们需要设置一个java Environment Variable。

According Hortonworks we need to set the client_jaas_client to connect. 根据Hortonworks,我们需要将client_jaas_client设置为connect。

The solution was using Py4j to call the Kafka Producer in JVM. 解决方案是使用Py4j在JVM中调用Kafka Producer。

See the example here . 请参阅此处的示例。

If the task is to solve this problem in python, another alternative could be to use confluent-kafka-python library that internally uses librdkafka that is written in C, and supports SASL, and the use of the keytab file. 如果任务是在python中解决这个问题,另一种选择可能是使用confluent-kafka-python库,它在内部使用用C编写的librdkafka ,并支持SASL,以及keytab文件的使用。 That wouldn't require having a separate Java process for the communication with kafka over SASL. 这不需要单独的Java进程来通过SASL与kafka进行通信。

For instructions also refer to the documentation of the librdkafka library: 有关说明,请参阅librdkafka库的文档:

https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka - general intro https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md - the properties that can be passed into the constructor of confluent_kafka.Producer and confluent_kafka.Consumer in python https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka - 一般介绍https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md - 可以传递的属性到python中的confluent_kafka.Producerconfluent_kafka.Consumer的构造函数

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM