Execute the scala script through spark-shell in silent mode

Question

Need to execute the scala script through spark-shell with silent mode. When I am using spark-shell -i "file.scala" , after the execution, I am getting into the scala interactive mode. I don't want to get into there.

I have tried to execute the spark-shell -i "file.scala". But I don't know how to execute the script in silent mode.

spark-shell -i "file.scala"

after execution, I get into

scala>

I don't want to get into the scala> mode

Updating (October 2019) for a script that terminates

This question is also about running a script that terminates, that is, a "scala script" that run by spark-shell -i script.scala > output.txt that stopts by yourself (internal instruction System.exit(0) terminates the script).
See this question with a good example .

It also needs a "silent mode" , it is expected to not pollute the output.txt .

Suppose Spark v2.2+ .

---

PS: there are a lot of cases (typically small tools and module/algorithm tests) where Spark interpreter can be better than compiler ... Please, "let's compile!" is not an answer here.

Answer 1

spark-shell -i file.scala keeps the interpreter open in the end, so System.exit(0) is required to be at the end of your script. The most appropriate solution is to place your code in try {} and put System.exit(0) in finally {} section.

If logging is requiered you can use something like this:

spark-shell < file.scala > test.log 2>&1 &

If you have limitations on editing file and you can't add System.exit(0) , use:

echo :quit | scala-shell -i file.scala

UPD

If you want to suppress everything in output except printlns you have to turn off logging for spark-shell. The sample of configs is here . Disabling any kind of logging in $SPARK-HOME/conf/log4j.properties should allow you to see only pritnlns. But I would not follow this approach with printlns. Using general Logging with log4j should be used instead of printlns. You can configure it so obtain the same results as with printlns. It boils down to configuring a pattern. This answer provides an example of a pattern that solves your issue.

Answer 2

最好的方法绝对是将您的 Scala 代码编译为 jar 并使用spark-submit但如果您只是在寻找快速迭代循环，您可以在解析您的 Scala 代码后简单地发出:quit ：

echo :quit | scala-shell -i yourfile.scala

Answer 3

Adding onto @rluta's answer. You can place the call to spark-shell command inside a shell script. Say the below in a shell script:

spark-shell < yourfile.scala

But this would require you to keep the lines of code within a line in case a statement is written on different lines.

OR

echo :quit | spark-shell -i yourfile.scala

This should