ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: scala/Function0$class

Question

I am trying to submit a spark job to a hadoop-yarn cluster via Apache Livy . The cluster is setup with steps specified here .

Java code is being run via IntelliJ on windows local machine. The spark and hadoop cluster is on linux servers. Other applications(w/o Livy) run perfectly fine with the operations on hdfs and spark computation.

I am trying to run the The error log I see in stderr of the application at the cluster :

INFO yarn.ApplicationMaster: Waiting for spark context initialization...
INFO driver.RSCDriver: Connecting to: master:10000
INFO driver.RSCDriver: Starting RPC server...
INFO rpc.RpcServer: Connected to the port 10001
WARN rsc.RSCConf: Your hostname, master, resolves to a loopback address, but we couldn't find any external IP address!
WARN rsc.RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
INFO driver.RSCDriver: Received job request 37e4684d-9de2-4a4b-9506-0b10a3e78a51
INFO driver.RSCDriver: SparkContext not yet up, queueing job request.
ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: scala/Function0$class
java.lang.NoClassDefFoundError: scala/Function0$class
    at org.apache.livy.shaded.json4s.ThreadLocal.<init>(Formats.scala:311)
    at org.apache.livy.shaded.json4s.DefaultFormats$class.$init$(Formats.scala:318)
    at org.apache.livy.shaded.json4s.DefaultFormats$.<init>(Formats.scala:296)
    at org.apache.livy.shaded.json4s.DefaultFormats$.<clinit>(Formats.scala)
    at org.apache.livy.repl.Session.<init>(Session.scala:66)
    at org.apache.livy.repl.ReplDriver.initializeSparkEntries(ReplDriver.scala:41)
    at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:333)
    at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)
Caused by: java.lang.ClassNotFoundException: scala.Function0$class
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 13 more
INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User class threw exception: java.lang.NoClassDefFoundError: scala/Function0$class
    at org.apache.livy.shaded.json4s.ThreadLocal.<init>(Formats.scala:311)
    at org.apache.livy.shaded.json4s.DefaultFormats$class.$init$(Formats.scala:318)
    at org.apache.livy.shaded.json4s.DefaultFormats$.<init>(Formats.scala:296)
    at org.apache.livy.shaded.json4s.DefaultFormats$.<clinit>(Formats.scala)
    at org.apache.livy.repl.Session.<init>(Session.scala:66)
    at org.apache.livy.repl.ReplDriver.initializeSparkEntries(ReplDriver.scala:41)
    at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:333)
    at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)
Caused by: java.lang.ClassNotFoundException: scala.Function0$class
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 13 more
)

The java code for submitting spark job :

URI uri = new URI("http", "username:password" , "host" , 8998, "", null, null);

Map<String,String> config = new HashMap<>();
config.put("spark.app.name","livy-poc");
config.put("livy.client.http.connection.timeout", "180s");
config.put("spark.driver.memory", "1g");

LivyClient client = new LivyClientBuilder(true).setURI(uri).setAll(config).build();

try {
    client.addJar(new URI("/path_to_jars/spark-core_2.12-2.4.2.jar")).get();
    client.addJar(new URI("/path_to_jars/scala-library-2.12.8.jar")).get();
    client.addJar(new URI("/path_to_jars/ThisJavaCode.jar")).get();

    System.out.printf("Running PiJob with %d samples...\n", 2);
    double pi = client.submit(new PiJob(2)).get();
    System.out.println("Pi is roughly: " + pi);
} catch (InterruptedException | ExecutionException e) {
    e.printStackTrace();
} finally {
    client.stop(true);
}
}

The livy.conf file has :

# What spark master Livy sessions should use.
livy.spark.master = yarn
# What spark deploy mode Livy sessions should use.
livy.spark.deployMode = cluster

Can some one point please out if i'm missing anything?

Answer 1

Livy appears to only support Spark releases built against Scala 2.11.x. See https://issues.apache.org/jira/browse/LIVY-423 .

Change your client.addJar(... lines to include a Scala 2.11 release and a Spark distro built against 2.11.