NoClassDefFoundError: org/apache/spark/SparkConf

Question

I have a simple Spark application and I for the life of me cannot run the output jar. I'm simply running mvn clean install and running the jar with java -jar SparkUdemy2-1.0-SNAPSHOT.jar

Below I've attached both the maven file and the small code snippet.

I made sure that the deps exist within my local m2. What is happening? It's imported with no issues.

Maven

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>SparkUdemy2</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>15.0</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>2.4.5</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>2.0.0</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.2.0</version>
            <scope>compile</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.0.2</version>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>com.learning.SparkMain</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Code

package com.learning;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import scala.Tuple2;

import java.util.Arrays;

public class SparkMain {
  public static void main(String[] args) {
    // Configure spark in local cluster - use all available cores available on machine
    // Without this, the application would be running on a single thread
    SparkConf conf = new SparkConf().setAppName("LearningSpark");
    JavaSparkContext sc = new JavaSparkContext(conf);

    JavaRDD<String> initialRDD = sc.textFile("s3n://s3-spark-data-bucket/input.txt");
    JavaPairRDD<Long, String> dat =
        initialRDD
            .map(sentence -> sentence.replaceAll("[^a-zA-Z\\s]", ""))
            .filter(line -> line.trim().length() > 0)
            .flatMap(line -> Arrays.asList(line.split(" ")).iterator())
            .mapToPair(word -> new Tuple2<>(word, 1L))
            .reduceByKey((v1, v2) -> v1 + v2)
            .mapToPair(tuple -> new Tuple2<>(tuple._2, tuple._1))
            .sortByKey(false);

    dat.foreach(item -> System.out.println(item));

    sc.close();
  }
}

Answer 1

Java does not know what dependencies it has to include in the classpath when you run a jar file. That's why you have to tell Maven to create a so-called fatjar (a jar that consists of your classes and all of the dependencies) using the shade plugin .

Answer 2

try changing your build in pom to following:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.5.1</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <executions>
                <execution>
                    <id>anything</id>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>