Unable to launch spark-shell on Spark 1.4.1b Cluster

Question

I'm unable to launch spark-shell on my Spark 1.4.1

This is launched on EMR instance. Last time when I launched Spark 1.4 I was able to launch spark-shell. I'm not sure why its failing this time.

Any suggestions are welcome.

Log when I launch spark-shell from terminal:

`

[hadoop@ip-xx-xxx-xxx-xxx ~]spark-shell
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
15/11/06 18:30:17 INFO spark.SecurityManager: Changing view acls to: hadoop
15/11/06 18:30:17 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/11/06 18:30:17 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/11/06 18:30:17 INFO spark.HttpServer: Starting HTTP Server
15/11/06 18:30:17 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/11/06 18:30:17 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:50579
15/11/06 18:30:17 INFO util.Utils: Successfully started service 'HTTP class server' on port 50579.
error: error while loading <root>, zip file is empty

Failed to initialize compiler: object scala.runtime in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programatically, settings.usejavacp.value = true.
15/11/06 18:30:17 WARN repl.SparkILoop$SparkILoopInterpreter: Warning: compiler accessed before init set up.  Assuming no postInit code.

Failed to initialize compiler: object scala.runtime in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programatically, settings.usejavacp.value = true.
Exception in thread "main" java.lang.AssertionError: assertion failed: null
    at scala.Predef$.assert(Predef.scala:179)
    at org.apache.spark.repl.SparkIMain.initializeSynchronous(SparkIMain.scala:247)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
    at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
    at org.apache.spark.repl.Main$.main(Main.scala:31)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/11/06 18:30:17 INFO util.Utils: Shutdown hook called
15/11/06 18:30:17 INFO util.Utils: Deleting directory /tmp/spark-3499c189-36cf-48b0-9c6f-fa97b8d89f86

`

Answer 1

Looks like a path issue. Is the location of your Scala script in your PATH environment variable?

In .bash_profile or .bashrc:

export PATH=$PATH:/path/to/scala

It's surprising that the Scala install didn't add it to /usr/local/bin though.