spark maven jar file dependency

Question

I have used Eclipse with Maven, Java and Scala to develop a simple Spark application. Under the target folder, Maven install command has created two JAR files one named as application.jar and the other one named as application-with-dependencies.jar . My question is which JAR file I should use to submit it to the Spark cluster? Also, why there are two JAR files? The file called application.jar is fraction of the size application-with-dependencies.jar

Answer 1

Maven usually generates a JAR file containing only your compiled classes and resources, but using some plugins, like maven-assembly-plugin , maven can also generate an uber-jar, ie: a JAR file that contains not only your classes and resources, but also libraries your project depends on.

Without more information application-with-dependencies-jar looks like one of those uber-jar.

From spark documentation looks like that's the JAR you must submit to the Spark cluster.