[英]Python Kafka Producer and Consumer in Cloudera Cluster
我有一个cloudera集群,在3台不同的计算机上有3个Broker。 我正在从集群内部的第四个开发。
我已经按如下方式创建了主题:创建主题/ usr / bin / kafka-topics --zookeeper主机:2181,host2:2181,hosts3:2181 / kafka --create --partitions 10 --replication-factor 2 --topic主题名称
我在zookeeper中的根目录不是root,它是/ kafka
这是我的生产者代码:
class Kafkaproducer(object):
def __init__(self, **kwargs):
if kwargs:
try:
self.producer = KafkaProducer(**kwargs)
except Exception as ex:
print "unable to create Producer Object " + str(ex)
self.iw = Imageworker()
log = Logger()
self.logs = log.logger('Producer')
def set_topic(self, topic):
"""
Set Topic for Producer
:param self:
:param topic: Topic String for Kafka
:return: no value
"""
self.topic = topic
print self.producer.partitions_for(topic )
def send_message(self, file):
"""
send a single message to kafka broker
:param self:
:param file: absolute filepath from file to send to broker
:return: no value
"""
print self.topic
try:
print "create json message .. "
message = self.iw.read_image_file(file)
except Exception as ex:
print "unable to read file" + str(ex)
try:
print "send message"+ self.iw.get_imagename(file)
self.producer.send(self.topic, message)
except Exception as Ex:
print "unable to send kafka message " + str(ex)
def _handle_fetch_response(self):
print "error"
def send_message_synchron(self, file ):
"""
:param data:
:return:
"""
try:
print "create json message .. "
message = self.iw.read_image_file(file)
except Exception as ex:
print "unable to read file" + str(ex)
try:
#print "send message "+ self.iw.get_imagename(file)
future = self.producer.send(self.topic, message)
future.error_on_callbacks=True
#result = future.get(timeout=1000)
result = future.succeeded()
print future.is_done
if result:
print future.value
print result
print "success!!!"
meta = future.get(timeout=100)
except Exception as ex:
print "unable to send kafka message " + str(ex)
try:
if future.is_done:
print "Message send successful "
except KafkaError:
log.exception()
print "Error in Kafka"
pass
def flush_producer(self):
self.producer.flush()
我能够与send_messages函数异步发送消息。 我也从使用的主题中获得分区数。 问题是消息消失了。
我已经用我的python使用者和以下语句检查了两次:
/opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/lib/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list myhosts --topic topic_name
此外,我想使用同步功能发送消息以获取未来的结果。 在这里,我无法获得未来的结果。 行结果= future.get(timeout = 1000)失败。
希望有人在这种情况下有想法。 非常感谢,
约恩
找到了问题,但不知道如何解决。 我从属性文件中读取了生产者配置
bootstrap_servers=['h1:9092' ,'h2:9092','h3:9092']
api_version=(0,10)
value_serializer=str.encode
buffer_memory=200000000
retries=5
max_block_ms=10000
producer = Kafkaproducer(**dic) # do not work
roducer = Kafkaproducer(bootstrap_servers=['h1:9092' ,'h2:9092','h3:9092'],api_version=(0,10)... # works well
在消费者网站上,我可以与消费者合作= Kafkaconsumer(** dic)
修复了生产方调用后,同步错误战争也消失了。 但是,为什么我不能用字典称呼制作人呢?
-> {'retries':5,'max_block_ms':10000,'buffer_memory':200000000,'bootstrap_servers':['h1:9092','h2:9092','h3:9092'],'value_serializer': 'str.encode','api_version':(0,10)}
谢谢
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.