在Spark中使用Scala脚本

Question

We are planning to using Scala on Spark to make computations. Just want to know what is the best way to execute Scala in Spark ; Scala as Script (or) Scala as Application . Is there any advantage / disadvantage between these 2 methods?

As mentioned here it is possible to execute Scala as Script . I am trying to skip the compilation process using sbt so that I can use Scala as script just like we will use Python

Answer 1

I suppose you mean by scala as script the scala REPL comes with spark (spark-shell) and scala application is the standlaone appliction packaged by sbt or maven .

• Use scala shell ( spark-shell ) to test your algoriothm/implementation . So it should be used as staging phase.
• When you have tested your implementation you should put it in a standalone application, package it and deliever a "fat jar" that could be submited by using spark-submit

Hope this is clear enough