启动并提交工作火花

Question

I try to "run" spark jobs width my java application, searching, I found the following two methods: ClientsArguments and SparkLauncher .

Could someone explain to me the difference between the two? The difference between launch and submit job/application Spark? Thank you.

Answer 1

SparkLauncher is just a wrapper library for spark-submit , it coverts your SparkLauncher code to spark-submit script and then trigger the jobs.

The mechanism is the same with spark-submit script, if you look at the source code of SparkLauncher, it uses ProcessBuilder to construct the shell.

If you want to use SparkLauncher , you need to specify $JAVA_HOME , $SPRAK_HOME and other essential parameters. There are some limitation for SparkLauncher , the machine your SparkLauncher runs must have a $JAVA_HOME and $SPARK_HOME (Spark Library) which is used for SparkLauncher locating the script and related dependencies. This is soft of impossible for some cloud environment like CloudFoundry etc.

You could assume that SparkLauncher equals spark-submit script, you could choose client or master, local or yarn mode.

ClientsArguments is a class for YARN script, which is only works for yarn-mode.