正确的Spark API Maven依赖关系

Question

I am wondering if the following dependency declarations are not enough to get access to the following class

org.apache.spark.api.java.function.PairFunction

Error at runtime

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/PairFunction
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:278)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.PairFunction
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

Dependencies declared

  <dependencies>
  <dependency>
     <groupId>org.scala-lang</groupId>
     <artifactId>scala-library</artifactId>
     <version>2.12.1</version>
   </dependency>
   <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>2.1.0</version>
</dependency>
   <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

I am running the .jar like so:

hadoop jar target/secondarysortspark-1.0-SNAPSHOT.jar   ~/projects/secondarysortspark/secondarysortspark/src/main/java/com/tom/secondarysortspark/data.txt

Thanks

Answer 1

For your spark version the scala version is not ok. I have the following properties and it works

<properties>
        <scala.tools.version>2.10</scala.tools.version>
        <scala.version>2.10.4</scala.version>
        <spark.version>1.6.1</spark.version>
    </properties>

At least scala 2.10 is required, as for spark-core_2.10 declare on your spark core dependency(take a look here spark )

Answer 2

The interface PairFunction is in spark-core so your declared dependencies are fine.

https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/api/java/function/PairFunction.java

The issue is that spark-core is not found on classpath during runtime.

The problem can be solved in a variety of ways depending on your set-up but the easiest in your case is likely to just pass the jar using -libjars option of your hadoop command.

Try something like

hadoop jar target/secondarysortspark-1.0-SNAPSHOT.jar ~/projects/secondarysortspark/secondarysortspark/src/main/java/com/tom/secondarysortspark/data.txt -libjars path/to/spark-core.jar

Let me know the results.

Answer 3

We have two options here, do a spark-submit and give the dependency jars in the command as -jars. Or else build a fat jar and do spark submit. Building the fat jar will fix the dependency issue.

Eg. spark-submit --master local -- class your_class jarfile_path --jars dependency_jars input_arguements