[英]Send Spark Streaming data back to the client
I am new to Apache Spark Streaming.我是 Apache Spark Streaming 的新手。 I am developing a spark streaming application to find the shortest path and sending the path again back to the client.
我正在开发一个火花流应用程序来找到最短路径并将路径再次发送回客户端。 I have written code for taking data and processing it with a function but I have an issue how can I send my results again back to the client Here is my code:
我已经编写了用于获取数据并使用 function 处理数据的代码,但是我有一个问题,如何将结果再次发送回客户端这是我的代码:
import networkx as nx
from pyspark import SparkConf,SparkContext
from pyspark.streaming import StreamingContext
TCP_IP = "127.0.0.1"
TCP_PORT = 5000
# Creating a Spark Configuration
conf=SparkConf()
conf.setAppName('ShortestPathApp')
sc= SparkContext(conf)
ssc= StreamingContext(sc,2)
def shortestPath(line):
# get the values from rdd
vehicleId = line[0]
source = line[1]
destination = line[2]
deadline = line[3]
# find shortest path
shortest = nx.dijkstra_path(G, source, destination)
# receive from Socket
dataStream =ssc.socketTextStream(TCP_IP,TCP_PORT)
vehicle_data = dataStream.map(lambda line: line.split(" "))
vehicle_data.foreachRDD(lambda rdd: rdd.foreach(shortestPath))
ssc.start()
ssc.awaitTermination()
How can I send the data back to the client如何将数据发送回客户端
Use StreamingContext
to push output data back to destination as a stream.使用
StreamingContext
将 output 数据作为 stream 推送回目的地。 you can create it as below.您可以按如下方式创建它。 Add method to get singleton SparkSession instance
添加方法获取 singleton SparkSession 实例
# Lazily instantiated global instance of SparkSession
def getSparkSessionInstance(sparkConf):
if ("sparkSessionSingletonInstance" not in globals()):
globals()["sparkSessionSingletonInstance"] = SparkSession \
.builder \
.config(conf=sparkConf) \
.getOrCreate()
return globals()["sparkSessionSingletonInstance"]
sparkSess = getSparkSessionInstance(rdd.context.getConf())
vehicle_data_df = sparkSess.createDataFrame(vehicle_data)
vehicle_data_df.writeStream\
.format("socket")\
.option("host",TCP_OUTPUT_IP) //Output socket IP address
.option("port",TCP_OUTPUT_PORT) //Output socket port
.outputMode('append')\
.start()\
.awaitTermination()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.