How to add offline Apache Spark dependencies to Java application?

Question

I want to get started with Apache Spark in Java. So I started following this tutorial . In the tutorial, they have shown a simple desktop app written in Eclipse IDE and compiled and run using the Gradle plugin for Eclipse.

I followed this tutorial, and everything worked perfectly. Here is their build.gradle file:

apply plugin: 'java-library'

repositories {
    jcenter()
}

dependencies {
    compileOnly 'org.apache.spark:spark-core_2.11:2.1.0'
    testImplementation 'org.apache.spark:spark-core_2.11:2.1.0','junit:junit:4.12'
}

So far so good! BUT now my real need was to be able to compile and run Spark applications in an offline system. So what I did was that I searched for org.apache.spark:spark-core_2.11:2.1.0 and junit:junit:4.12 in jcenter and downloaded all the .jar files and then created a folder in my project root named libs and then added all those .jar files to my project's Build Path by following this method .

And I modified the build.gradle file like this:

apply plugin: 'java'
apply plugin: 'application'

//apply plugin: 'java-library'


repositories {
    jcenter()
}

dependencies {
    compile fileTree(dir: 'libs', include: '*.jar')
}

When I compile this modified application, I get the following errors in the console. The question is that what am I doing wrong? How to I get rid of the problem?

Working Directory: /home/jason/WorkspaceOfGetStartedWithSparkJava/FirstApacheSparkProjectWithOfflineDependencies
Gradle User Home: /usr/share/gradle/bin
Gradle Distribution: Gradle wrapper from target build
Gradle Version: 4.3
Java Home: /usr/lib/jvm/java-8-oracle
JVM Arguments: None
Program Arguments: None
Build Scans Enabled: false
Offline Mode Enabled: false
Gradle Tasks: build

:compileJava/home/jason/WorkspaceOfGetStartedWithSparkJava/FirstApacheSparkProjectWithOfflineDependencies/src/main/java/main_package/SparkDriverProgram.java:21: error: cannot access Cloneable
        conf.setAppName("Schneider");
            ^
  class file for scala.Cloneable not found
/home/jason/WorkspaceOfGetStartedWithSparkJava/FirstApacheSparkProjectWithOfflineDependencies/src/main/java/main_package/SparkDriverProgram.java:22: error: cannot access Serializable
        conf.setMaster("local");
            ^
  class file for scala.Serializable not found
2 errors
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s
1 actionable task: 1 executed

Answer 1

SparkConf implements the interfaces scala.Cloneable and scala.Serializable both of which are not part of org.apache.spark:spark-core but of org.apache.spark:scala-library. If you add that jar to your project the same way your did with the core jar those two errors should disappear.