[英]Python-Kafka: Keep polling topic infinitely
I am using python-kafka to listen to a kafka topic and use that the records.我正在使用 python-kafka 来收听一个 kafka 主题并使用该记录。 I want to keep it polling infinitely without any exit.
我想让它无限轮询而没有任何退出。 This is my code below:
这是我下面的代码:
def test():
consumer = KafkaConsumer('abc', 'localhost:9092', auto_offset_reset='earliest')
for msg in consumer:
print(msg.value)
This code just reads the data and exits directly.这段代码只是读取数据并直接退出。 Is there a way to keep listening to topics even if message is not pushed to it?
即使没有将消息推送到主题,有没有办法继续收听主题?
Any relevant example where the topic is continuously monitored is also great for me.任何持续监控主题的相关示例对我来说也很棒。
Using confluent_kafka
使用
confluent_kafka
import time
from confluent_kafka import Consumer
consumer = Consumer({
'bootstrap.servers': 'localhost:9092',
'group.id': 'my-consumer-1',
'auto.offset.reset': 'earliest'
})
consumer.subscribe(['topicName'])
while True:
try:
message = consumer.poll(10.0)
if not message:
time.sleep(120) # Sleep for 2 minutes
if message.error():
print(f"Consumer error: {message.error()}")
continue
print(f"Received message: {message.value().decode('utf-8')}")
except:
# Handle any exception here
...
finally:
consumer.close()
print("Goodbye")
Using kafka-python
使用
kafka-python
import time
from kafka import KafkaConsumer
consumer = KafkaConsumer(
bootstrap_servers=['localhost:9092'],
auto_offset_reset='earliest',
group_id='my-consumer-1',
)
consumer.subscribe(['topicName'])
while True:
try:
message = consumer.poll(10.0)
if not message:
time.sleep(120) # Sleep for 2 minutes
if message.error():
print(f"Consumer error: {message.error()}")
continue
print(f"Received message: {message.value().decode('utf-8')}")
except:
# Handle any exception here
...
finally:
consumer.close()
print("Goodbye")
for kafka-python following solution worked对于 kafka-python 以下解决方案有效
from kafka import KafkaConsumer
def consume_message(topic_name):
consumer = KafkaConsumer(
bootstrap_servers=['localhost:9092'], auto_offset_reset='earliest'
)
consumer.subscribe(topic_name)
while True:
try:
records = consumer.poll(timeout_ms=1000)
for topic_data, consumer_records in records.items():
for consumer_record in consumer_records:
print("Received message: " + str(consumer_record.value.decode('utf-8')))
continue
except Exception as e:
print(e)
continue
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.