java.lang.NoClassDefFoundError when running a spark jar application

Question

I compiled my Apache Spark application which is written in Scala with sbt inside the IntellijIDEA and works fine when running inside the IntelliJ. But when I do compile and package it as a jar file and run it on a remote server, I got this error, when the code reaches where I try to create an Envelope instance inside the org/locationtech/jts/geom/Envelope

Exception in thread "main" java.lang.NoClassDefFoundError: org/locationtech/jts/geom/Envelope
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1629)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:79)
at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:520)...

I understand that this is the problem of inconsistency between versions of libraries and also know that NoClassDefFoundError means that the library was accessible while compiling and is not accessible in runtime, but I can not solve the problem.

This is my build.sbt file:

name := "MySpark"
version := "0.1"
scalaVersion := "2.11.6"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.3.0", "org.locationtech.jts" % "jts-core" % "1.16.0" 
)

The versions of Spark and Scala on the remote computer are the same as in the build.sbt file.

When I run the evicted in the sbt shell I get this info, but I don't how to use this info to solve my problem.

[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn]  * io.netty:netty:3.9.9.Final is selected over {3.6.2.Final, 3.7.0.Final}
[warn]      +- org.apache.spark:spark-core_2.11:2.3.0             (depends on 3.9.9.Final)
[warn]      +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.6.2.Final)
[warn]      +- org.apache.hadoop:hadoop-hdfs:2.6.5                (depends on 3.6.2.Final)
[warn]  * commons-net:commons-net:2.2 is selected over 3.1
[warn]      +- org.apache.spark:spark-core_2.11:2.3.0             (depends on 2.2)
[warn]      +- org.apache.hadoop:hadoop-common:2.6.5              (depends on 3.1)
[warn]  * com.google.guava:guava:11.0.2 is selected over {12.0.1, 16.0.1}
[warn]      +- org.apache.hadoop:hadoop-yarn-client:2.6.5         (depends on 11.0.2)
[warn]      +- org.apache.hadoop:hadoop-yarn-api:2.6.5            (depends on 11.0.2)
[warn]      +- org.apache.hadoop:hadoop-yarn-common:2.6.5         (depends on 11.0.2)
[warn]      +- org.apache.hadoop:hadoop-yarn-server-nodemanager:2.6.5 (depends on 11.0.2)
[warn]      +- org.apache.hadoop:hadoop-yarn-server-common:2.6.5  (depends on 11.0.2)
[warn]      +- org.apache.hadoop:hadoop-hdfs:2.6.5                (depends on 11.0.2)
[warn]      +- org.apache.curator:curator-framework:2.6.0         (depends on 16.0.1)
[warn]      +- org.apache.curator:curator-client:2.6.0            (depends on 16.0.1)
[warn]      +- org.apache.curator:curator-recipes:2.6.0           (depends on 16.0.1)
[warn]      +- org.apache.hadoop:hadoop-common:2.6.5              (depends on 16.0.1)
[warn]      +- org.htrace:htrace-core:3.0.4                       (depends on 12.0.1)
[warn] Run 'evicted' to see detailed eviction warnings
[info] Here are other dependency conflicts that were resolved:
[info]  * com.thoughtworks.paranamer:paranamer:2.8 is selected over {2.6, 2.3}
[info]      +- com.fasterxml.jackson.module:jackson-module-paranamer:2.7.9 (depends on 2.8)
[info]      +- org.json4s:json4s-core_2.11:3.2.11                 (depends on 2.6)
[info]      +- org.apache.avro:avro:1.7.7 

Answer 1

If you run the sbt package cmd, the generated jar will only containe your source files but no 3rd party library.

So If you deploy your Jar and try to run it. Your runtime environment must provide the external librairies specified here

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.3.0", "org.locationtech.jts" % "jts-core" % "1.16.0" 
)

As you run a Spark application "org.apache.spark" %% "spark-core" % "2.3.0" is present in you runtime environment classpath. the spark-submit will do it for you. "org.locationtech.jts" % "jts-core" % "1.16.0" is a specific 3rd party lib and by default, I don't think you runtime environment will include it.

You have to use a plugin :

• sbt-assemby
• sbt-onejar
• sbt-native-packager

and configure it to include your 3rd party librairies. So your application will come with the necessary classes according to the chosen solution. In this case, I strongly advise you to add the scope provided to the spark libs already present in your runtime environment. It should exclude them from the package and prevent potential conflicts between your package and what already exists on your target (runtime) environment

Answer 2

尝试使用--packages将依赖项添加到spark-submit命令中

spark-submit --packages org.locationtech.jts:jts-core:1.16.0 --class labX.applications.ServerTest --executor-memory 2048m --executor-cores 1 --num-executors 17 mySparkApp_2.11-0.1.jar