Spark-shell uses different version of Scala. Used homebrew to install both scala and apache-spark
Question
I installed scala and apache-spark using homebrew and it installed scala 2.12.4 and apache-spark 2.2.0. However, if you checkout spark-shell -version it uses a different scala version.
How do I set spark-shell to use the installed scala version? Is there way to set scala version used by apache-spark during homebrew installation?
1 answers
solution1
1 ACCPTED 2017-10-24 02:42:02
TL;DR You cannot.
Two problems:
- Spark (it is not really specific to Spark) will use Scala version which has been used to compile it. Version of Scala compile installed on the machine is not relevant at all.
- Spark doesn't support Scala 2.12 yet, so recompiling is not an option.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
How to deploy scala files used in spark-shell on cluster?
Exiting Spark-shell from the scala script
Where is scala on node with spark-shell installed?
Registering complex scala classes with Kryo in spark-shell and Scala jars
scala code works on spark-shell but not in spark-submit
Accessing a specific element of an Array RDD in apache-spark scala
Change datatype in Array[StructField] at an index spark-shell/ scala
How to import own scala package using spark-shell?
How to read strings from scala spark-shell Array[String]
Spark-Shell Scala XML how to concatenate attributes
Related Tags