[英]Kafka Consumer didn't receiving any messages from its Producer
以下是我为kafka生产者编写的python编码,我不确定消息是否可以发布到Kafka Broker。 因为消费者方没有收到任何消息。 我的使用者python程序在使用生产者控制台命令进行测试时运行良好。
from __future__ import print_function
import sys
from pyspark import SparkContext
from kafka import KafkaClient, SimpleProducer
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage:spark-submit producer1.py <input file>", file=sys.stderr)
exit(-1)
sc = SparkContext(appName="PythonRegression")
def sendkafka(messages):
## Set broker port
kafka = KafkaClient("localhost:9092")
producer = SimpleProducer(kafka, async=True, batch_send_every_n=5,
batch_send_every_t=10)
send_counts = 0
for message in messages:
try:
print(message)
## Set topic name and push messages to the Kafka Broker
yield producer.send_messages('test', message.encode('utf-8'))
except Exception, e:
print("Error: %s" % str(e))
else:
send_counts += 1
print("The count of prediction results which were sent IN THIS PARTITION
is %d.\n" % send_counts)
## Connect and read the file.
rawData = sc.textFile(sys.argv[1])
## Find and skip the first row
dataHeader = rawData.first()
data = rawData.filter(lambda x: x != dataHeader)
## Collect the RDDs.
sentRDD = data.mapPartitions(sendkafka)
sentRDD.collect()
## Stop file connection
sc.stop()
这是我的“消费者” python编码
from __future__ import print_function
import sys
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
if len(sys.argv) < 3:
print ("Program to pulls the messages from kafka brokers.")
print("Usage: consume.py <zk> <topic>", file=sys.stderr)
else:
## Flow
## Loads settings from system properties, for launching of spark-submit.
sc = SparkContext(appName="PythonStreamingKafkaWordCount")
## Create a StreamingContext using an existing SparkContext.
ssc = StreamingContext(sc, 10)
## Get everything after the python script name
zkQuorum, topic = sys.argv[1:]
## Create an input stream that pulls messages from Kafka Brokers.
kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer",
{topic: 1})
##
lines = kvs.map(lambda x: x[1])
## Print the messages pulled from Kakfa Brokers
lines.pprint()
## Save the pulled messages as file
## lines.saveAsTextFiles("OutputA")
## Start receiving data and processing it
ssc.start()
## Allows the current process to wait for the termination of the context
ssc.awaitTermination()
我通常使用kafka-console-consumer(Apache Kafka的一部分)调试此类问题,以从您尝试制作的主题中使用。 如果控制台使用者收到消息,则您知道消息已到达Kafka。
如果首先运行生产者,让它完成,然后启动使用者,则问题可能是使用者从日志末尾开始并正在等待其他消息。 确保您首先启动使用者,或者将其配置为从头开始自动启动(抱歉,不确定如何使用Python客户端执行此操作)。
如果消息随着生产请求而增加,则可以检查主题中的消息数:
./bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
--broker-list <Kafka_broker_hostname>:<broker_port> --topic Que1 \
--time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
如果消息数量在增加,则表明生产者工作正常。
好吧,我认为我的本地Zookeeper或Kafka出了点问题,因为我在另一台服务器上对其进行了完美测试。 但是,感谢那些回答我的人;)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.