从bash启动spark作业，从文本文件获取参数

Question

I am running a standalone spark instance that I launch with:

/usr/local/spark-1.6.0/bin/spark-submit --class "run.Main" --conf spark.driver.userClassPathFirst=true --driver-memory 45G --jars $(echo /var/myapp/lib/*.jar | tr ' ' ',') mycoolapp.jar "local[6]" "parA" "parB" "parC" "parD"

what I do manually is to launch it for a specific "parA" value.

Then, once is finished, I relaunch it with a new value for "parA". I have all the possible "parA" values listed in a .txt files, and I am wondering if it's possible to write a bash script that does this for me, ie launching the script and automatically picking the next "parA" value from the text file.

Of course, I need that it waits to have finished a Spark job before launching the next, since I am using Spark on a single machine and each single job eats almost all the RAM on the machine...

any guidance on that is more than welcome.

Answer 1

Something like this. You just iterate over an array of arguments. And you no need to worry about how to wait until the end of the job, because submit operation is synchronous.

#!/bin/bash

declare -a parAs=('parA0' 'parA1' 'parA2')

for parA in "${parAs[@]}"; do
    echoString=$(eval echo /var/myapp/lib/*.jar | tr ' ' ',')
    ./bin/submit ....--jars $echoString...... $parA .....
done