java.lang.NoClassDefFoundError in spark-submit

Question

I've built the project from https://github.com/tribbloid/spookystuff/tree/branch-0.4 here using

MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" mvn package -DskipTests=true

from here https://gist.github.com/titipata/13fad88df5525d607f24

Then I'm using a fat-jar created in core/target/x-jar-with-dependencies.jar in my own separate project.

<dependency>
    <groupId>com.tribbloid.spookystuff</groupId>
    <artifactId>spookystuff-core_2.10</artifactId>
    <scope>system</scope>
    <version>0.4.0-SNAPSHOT</version>
    <systemPath>${basedir}/lib/x-jar-with-dependencies.jar</systemPath>
</dependency>

And I'm also creating fat-jar of this project I'm using spookystuff in ( y-jar-with-dependencies.jar lets say).

Now when running

spark-submit --class org.webcrawler.core.Driver --master local[*] /path/to/my/y-jar-with-dependencies.jar

I get an error stating

Exception in thread "main" java.lang.NoClassDefFoundError: com/tribbloids/spookystuff/actions/Action

Investigation:

jar -tvf /path/to/my/x-jar-with-dependencies.jar | grep "com/tribbloids/spookystuff/actions/Action"

I get results but

jar -tvf /path/to/my/y-jar-with-dependencies.jar | grep "com/tribbloids/spookystuff/actions/Action"

I get no classes found. Where am I wrong in all this?

Answer 1

在Ykyuen发布的https://eureka.ykyuen.info/2014/06/10/maven-include-system-scope-dependency-in-maven-assembly-plugin/之后，我解决了我的问题。