[英]confluent-kafka based consumer in Python does not work
Very new to kafka and Avro. 对kafka和Avro来说是新手。 I am stuck with a problem and can not seem to figure out what is going wrong here.
我遇到了一个问题,似乎无法弄清楚这里出了什么问题。 I had written a producer and consumer of kafka which uses Avro as the serialization format.
我已经写过使用avro作为序列化格式的kafka的生产者和消费者。 The producer code is working properly.
生产者代码正常运行。 As after running that code when I run the
kafka-avro-console-consumer
it give me as following - 在运行该代码后,当我运行
kafka-avro-console-consumer
它显示如下:
bin/kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic test --property schema.registry.url=http://127.0.0.1:8081 --from-beginning
{"name":{"string":"Hello World!"}}
{"name":{"string":"Hello World!"}}
{"name":{"string":"Hello World!"}}
However, when I try to do the same using python (following this the most basic example ) I write the following code - 但是,当我尝试使用python做同样的事情时(以下是最基本的示例 ),我编写了以下代码-
from confluent_kafka import KafkaError
from confluent_kafka.avro import AvroConsumer
from confluent_kafka.avro.serializer import SerializerError
class AvroConsumerAdapter(object):
def __init__(self, topic='test'):
self.topic = topic
self.consumer = AvroConsumer({'bootstrap.servers': 'localhost:9092',
'schema.registry.url': 'http://127.0.0.1:8081',
'group.id': 'mygroup'})
self.consumer.subscribe([topic])
def start_consuming(self):
running = True
while running:
try:
msg = self.consumer.poll(10)
if msg:
print(msg.value())
if not msg.error():
print("Here - 1")
print(msg.value())
elif msg.error().code() != KafkaError._PARTITION_EOF:
print("here-2")
print(msg.error())
running = False
else:
print('Here-3')
print(msg.error())
except SerializerError as e:
print("Message deserialization failed for %s: %s" % (msg, e))
running = False
except Exception as ex:
print(ex)
running = False
self.consumer.close()
This client stays there forever and never prints anything. 该客户永远呆在那里,从不打印任何内容。 I am not sure what is wrong here.
我不确定这里出什么问题了。 Can anyone please help me in this.
谁能帮我这个忙。
Check out the topic config options -- you need to set auto.offset.reset': 'smallest'
if you want to process all of the data currently in the topic. 查看主题配置选项 -如果要处理主题中当前的所有数据,则需要设置
auto.offset.reset': 'smallest'
。 By default it's largest
which means it'll only show new rows of data produced. 默认情况下,它是
largest
,这意味着它只会显示产生的新数据行。 You can verify this by leaving your current Python code running and producing new messages to the topic - you should see the Python code pick them up. 您可以通过运行当前的Python代码并为该主题生成新消息来验证这一点-您应该看到Python代码将其选中。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.