[英]SparkStreaming, RabbitMQ and MQTT in python using pika
只是为了让事情变得棘手,我想使用来自rabbitMQ队列的消息。 现在我知道有一个针对兔子的MQTT插件( https://www.rabbitmq.com/mqtt.html )。
但是,我似乎无法在Spark消耗由pika生成的消息的情况下进行示例工作。
例如,我在这里使用简单的wordcount.py程序( https://spark.apache.org/docs/1.2.0/streaming-programming-guide.html ),看看我是否可以在下面看到一个消息生产者方式:
import sys
import pika
import json
import future
import pprofile
def sendJson(json):
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='analytics', durable=True)
channel.queue_bind(exchange='analytics_exchange',
queue='analytics')
channel.basic_publish(exchange='analytics_exchange', routing_key='analytics',body=json)
connection.close()
if __name__ == "__main__":
with open(sys.argv[1],'r') as json_file:
sendJson(json_file.read())
Sparkstreaming 消费者如下:
import sys
import operator
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.mqtt import MQTTUtils
sc = SparkContext(appName="SS")
sc.setLogLevel("ERROR")
ssc = StreamingContext(sc, 1)
ssc.checkpoint("checkpoint")
#ssc.setLogLevel("ERROR")
#RabbitMQ
"""EXCHANGE = 'analytics_exchange'
EXCHANGE_TYPE = 'direct'
QUEUE = 'analytics'
ROUTING_KEY = 'analytics'
RESPONSE_ROUTING_KEY = 'analytics-response'
"""
brokerUrl = "localhost:5672" # "tcp://iot.eclipse.org:1883"
topic = "analytics"
mqttStream = MQTTUtils.createStream(ssc, brokerUrl, topic)
#dummy functions - nothing interesting...
words = mqttStream.flatMap(lambda line: line.split(" "))
pairs = words.map(lambda word: (word, 1))
wordCounts = pairs.reduceByKey(lambda x, y: x + y)
wordCounts.pprint()
ssc.start()
ssc.awaitTermination()
但是,与简单的wordcount示例不同,我无法使其工作并出现以下错误:
16/06/16 17:41:35 ERROR Executor: Exception in task 0.0 in stage 7.0 (TID 8)
java.lang.NullPointerException
at org.eclipse.paho.client.mqttv3.MqttConnectOptions.validateURI(MqttConnectOptions.java:457)
at org.eclipse.paho.client.mqttv3.MqttAsyncClient.<init>(MqttAsyncClient.java:273)
所以我的问题是, MQTTUtils.createStream(ssc, brokerUrl, topic)
的设置应该是什么MQTTUtils.createStream(ssc, brokerUrl, topic)
以便监听队列以及是否有更丰富的示例以及这些示例如何映射到rabbitMQ的示例。
我正在运行我的消费者代码: ./bin/spark-submit ../../bb/code/skunkworks/sparkMQTTRabbit.py
/ ./bin/spark-submit ../../bb/code/skunkworks/sparkMQTTRabbit.py
我已按照以下注释建议的TCP参数更新了生产者代码:
url_location = 'tcp://localhost'
url = os.environ.get('', url_location)
params = pika.URLParameters(url)
connection = pika.BlockingConnection(params)
和火花流:
brokerUrl = "tcp://127.0.0.1:5672"
topic = "#" #all messages
mqttStream = MQTTUtils.createStream(ssc, brokerUrl, topic)
records = mqttStream.flatMap(lambda line: json.loads(line))
count = records.map(lambda rec: len(rec))
total = count.reduce(lambda a, b: a + b)
total.pprint()
看起来你使用了错误的端口号。 假如说:
rabbitmq-plugins enable rabbitmq_mqtt
)并重启RabbitMQ服务器 spark-streaming-mqtt
执行时spark-submit
/ pyspark
(或者与packages
或jars
/ driver-class-path
) 您可以使用TCP与tcp://localhost:1883
。 您还必须记住MQTT正在使用amq.topic
。
快速入门 :
使用以下内容创建Dockerfile
:
FROM rabbitmq:3-management RUN rabbitmq-plugins enable rabbitmq_mqtt
构建Docker镜像:
docker build -t rabbit_mqtt .
启动映像并等待服务器准备就绪:
docker run -p 15672:15672 -p 5672:5672 -p 1883:1883 rabbit_mqtt
使用以下内容创建producer.py
:
import pika import time connection = pika.BlockingConnection(pika.ConnectionParameters( host='localhost')) channel = connection.channel() channel.exchange_declare(exchange='amq.topic', type='topic', durable=True) for i in range(1000): channel.basic_publish( exchange='amq.topic', # amq.topic as exchange routing_key='hello', # Routing key used by producer body='Hello World {0}'.format(i) ) time.sleep(3) connection.close()
开始生产者
python producer.py
并访问管理控制台http://127.0.0.1:15672/#/exchanges/%2F/amq.topic
看到收到的消息。
使用以下内容创建consumer.py
:
from pyspark import SparkContext from pyspark.streaming import StreamingContext from pyspark.streaming.mqtt import MQTTUtils sc = SparkContext() ssc = StreamingContext(sc, 10) mqttStream = MQTTUtils.createStream( ssc, "tcp://localhost:1883", # Note both port number and protocol "hello" # The same routing key as used by producer ) mqttStream.count().pprint() ssc.start() ssc.awaitTermination() ssc.stop()
下载依赖项(将Scala版本调整为用于构建Spark和Spark版本的版本):
mvn dependency:get -Dartifact=org.apache.spark:spark-streaming-mqtt_2.11:1.6.1
确保SPARK_HOME
和PYTHONPATH
指向正确的目录。
提交consumer.py
with(像以前一样调整版本):
spark-submit --packages org.apache.spark:spark-streaming-mqtt_2.11:1.6.1 consumer.py
如果您按照所有步骤操作,则应在Spark日志中看到Hello world消息。
从MqttAsyncClient
Javadoc,服务器URI必须具有以下方案之一: tcp://
, ssl://
或local://
。 您需要更改上面的brokerUrl
以获得其中一个方案。
有关更多信息,请MqttAsyncClient
的源代码链接:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.