从 Windows 机器连接到 Spark

Question

I'm very (very!) new to Spark and Scala. I've been trying to implement what I thought to be the easy task of connecting to a linux machine that has Spark on it, and running a simple code.

When I create a simple Scala code, build a jar from it, place it in the machine and run spark-submit, everything works and I get a result. (like the "SimpleApp" example here: http://spark.apache.org/docs/latest/quick-start.html )

My question is: Are all of these steps mandatory? ? Must I compile, build and copy the jar to the machine and then manually run it every I change it?

Assume that the jar is already on the machine, is there a way to run it (calling spark-submit) directly from a different code through my IDE?

Taking it a bit further, if lets say I want to run different tasks, do I have to create different jars and place all of them on the machine? Are there any other approaches?

Any help will be appreciated! Thanks!

Answer 1

There are two modes of running your code either submitting your job to the server. or by running in local mode which requires no Spark Cluster to be setup. Most generally use this for building and testing their application on small data-sets and then build and submit the tasks as jobs for production.

Running in Local Mode

val conf = new SparkConf().setMaster("local").setAppName("wordCount Example")

Setting master as "local" spark along with your application.

If you have already Built you jars you can use the same by specifying the spark masters url and by adding the required jars you can submit the job to a remote cluster.

val conf = new SparkConf()
      .setMaster("spark://cyborg:7077")
      .setAppName("SubmitJobToCluster Example")
      .setJars(Seq("target/spark-example-1.0-SNAPSHOT-driver.jar"))

Using the spark conf you can initialize SparkContext in your application and use it either in a local or cluster setup.

 val sc = new SparkContext(conf)

This is a old project spark-examples you have samples programs which you can run directly from your IDE.

So Answering you questions

• Are all of these steps mandatory? ? Must I compile, build and copy the jar to the machine and then manually run it every I change it? NO
• Assume that the jar is already on the machine, is there a way to run it (calling spark-submit) directly from a different code through my IDE? Yes you can. The above example does it.
• Taking it a bit further, if lets say I want to run different tasks, do I have to create different jars and place all of them on the machine? Are there any other approaches? Yes You just need one jar containing all your tasks and dependencies you can specify the class while submitting the job to spark. When doing it pro-grammatically you have complete control over it.