NoClassDefFoundError while running Spark depending neo4j jar(scala)

Question

I have a program trying to connect to Neo4j database and run on Spark, testApp.scala , and I package it using sbt package to package it in a.jar with dependencies according to this_contribution (I already have the neo4j-spark-connector-2.0.0-M2.jar )

resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"
libraryDependencies += "neo4j-contrib" % "neo4j-spark-connector" % "2.0.0-M2"

However while I tried spark-submit --class "testApp" a.jar it turns out to be a NoClassDefFoundError

Exception in thread "main" java.lang.NoClassDefFoundError: org/neo4j/spark/Neo4j$ in the code val n = Neo4j(sc)

There are 2 more things I have to mention

1) I used jar vtf to check the content in a.jar , it only has testApp.class , no class of neo4j is in it, but the package process was success (does it mean the neo4j-spark-connector-2.0.0-M2.jar is not packaged in?)

2) I can use spark-shell --packages neo4j-contrib:neo4j-spark-connector:2.0.0-M2 and type the code in testApp.scala , there is no problem (eg the wrong line above is val n = Neo4j(sc) but it can work in spark-shell)

Answer 1

You may try using the --jars option with spark-submit . For example

./bin/spark-submit --class "fully-qualified-class-name" --master "master-url" --jars "path-of-your-dependency-jar"

or you can also use spark.driver.extraClassPath="jars-class-path" to solve the issue.Hope this helps.

Answer 2

As the content in the .jar does not contain Neo4j class, it is the packaging problem.

What we should modify is sbt , instead of sbt package , we should use sbt clean assembly instead. This helps create a .jar pack containing all the dependencies in it.

If you use only sbt package , the compile progress is ok, but it will not pack neo4j-*.jar into your .jar . So during the run time it throws an NoClassDefError