使用Maven的Apache Spark错误

Question

I'm currently trying to learn Apache Spark. I'm using Maven and Java, as I'm not familiar with Scala or SBT.

Also, all the examples I tried seem to have the "pom.xml" file with over 2000 lines. Are that many dependencies required?

Here's the example I tried. Here's my current "pom.xml" file. This is the error I'm getting:

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:666)
at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:657)
at misc.apache2.main(apache2.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

Process finished with exit code 1

Also, if I'm using a Maven project for Apache Spark, do I need to have a build.sbt file?

Currently running with IntelliJ 16.1.3, Windows 10.

Answer 1

As for what I see, you are using different versions of Spark.

From your pom.xml , you are using version 1.2.0 of Spark Core, version 2.0.0 of Spark Sql and version 1.6.1 of Spark Hive.

Try using the same version of Spark for all Spark dependencies:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.0.0-preview</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>2.0.0-preview</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.11</artifactId>
    <version>2.0.0-preview</version>
</dependency>