[英]What needs to be changed when we switch Spark from Standalone to Yarn-Client?
Currently we have a program which is a web service, receiving SQL queries and use SQLContext
to respond. 当前,我们有一个程序是一个Web服务,它接收SQL查询并使用SQLContext
进行响应。 The program is now in standalone mode, we set spark.master
to a specific URL. 该程序现在处于独立模式,我们将spark.master
设置为特定的URL。 The structure is something like below: 结构如下所示:
object SomeApp extends App
{
val conf = new SparkConf().setMaster("spark://10.21.173.181:7077")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
while(true)
{
val query = Listen_to_query()
val response = sqlContext.sql(query)
send(response)
}
}
Now we are going to shift the system to Spark on Yarn, and it seems that we should use submit
to submit jobs to yarn. 现在,我们将系统转换为“ Spark on Yarn”,看来我们应该使用submit
将作业提交到yarn。 It would be strange to deploy such a "service" on yarn which won't stop like ordinary "Jobs". 在纱线上部署这样的“服务”不会像普通的“作业”那样停止会很奇怪。 But we don't know how to separate "Jobs" from our program. 但是我们不知道如何从程序中分离“工作”。
Do you have any suggestions? 你有什么建议吗? Thank you! 谢谢!
So if you just want to submit your jobs to yarn you can just change the master param. 因此,如果您只想将工作提交给yarn,就可以更改主参数。 However it sounds like you are looking for a long running shared Spark Context and there are a few options for something like this. 但是,听起来您正在寻找长期运行的共享Spark Context,并且有一些类似的选项。 There is https://github.com/spark-jobserver/spark-jobserver and https://github.com/ibm-et/spark-kernel . 有https://github.com/spark-jobserver/spark-jobserver和https://github.com/ibm-et/spark-kernel 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.