I'm using the KafkaClient in python's pykafka. I'm trying to read a text file and produce its lines to a topic then read it by a consumer. However on running it only reads individual letters in the message not the words or the lines of the text file. What am I doing wrong?
my producer is
from pykafka import KafkaClient
text = open('filename.txt','r').read()
text = text.split()
client = KafkaClient(hosts='localhost:9099')
topic = client.topics['topic']
producer = topic.get_sync_producer()
for i in text:
producer.produce(i.encode('ascii'))
my consumer is
from pykafka import KafkaClient
client = KafkaClient(hosts='localhost:9099')
topic = client.topics['topic']
consumer = topic.get_simple_consumer()
for message in consumer:
if message is not None:
print(message.offset, message.value.decode())
Would appreciate some pointers. I'm wondering if this is the best way to read a text file and run it through kafka.
I got the desired output by switching libraries to the kafka libraries.
Producer
from kafka import KafkaProducer
producer =
KafkaProducer(bootstrap_servers='localhost:9099')
with open('filename.txt')
as f:
for line in f:
producer.send('topic',line.encode('ascii'))
producer.flush()
producer.close()
Consumer
from kafka import KafkaConsumer
consumer =
KafkaConsumer('topic',bootstrap_servers='localhost:9099')
for msg in consumer:
print(msg.value.decode())
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.