What needs to be changed when we switch Spark from Standalone to Yarn-Client?

Question

Currently we have a program which is a web service, receiving SQL queries and use SQLContext to respond. The program is now in standalone mode, we set spark.master to a specific URL. The structure is something like below:

object SomeApp extends App
{
    val conf = new SparkConf().setMaster("spark://10.21.173.181:7077")
    val sc = new SparkContext(conf)
    val sqlContext = new SQLContext(sc)

    while(true)
    {
        val query = Listen_to_query()
        val response = sqlContext.sql(query)
        send(response)
    }
}

Now we are going to shift the system to Spark on Yarn, and it seems that we should use submit to submit jobs to yarn. It would be strange to deploy such a "service" on yarn which won't stop like ordinary "Jobs". But we don't know how to separate "Jobs" from our program.

Do you have any suggestions? Thank you!

Answer 1

So if you just want to submit your jobs to yarn you can just change the master param. However it sounds like you are looking for a long running shared Spark Context and there are a few options for something like this. There is https://github.com/spark-jobserver/spark-jobserver and https://github.com/ibm-et/spark-kernel .