I am trying to connect to spark master on a remote system through java app
I am using
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.1</version>
</dependency>
and code
{
SparkSession sparkSession = SparkSession.builder().
master("spark://ip:7077")
.appName("spark session example")
.getOrCreate();
JavaSparkContext sc = new JavaSparkContext(sparkSession.sparkContext());
}
Getting
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:713)
at org.apache.spark.sql.SparkSession$Builder.master(SparkSession.scala:766)
at com.mobelisk.spark.JavaSparkPi.main(JavaSparkPi.java:9)
Also If I change to
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
**<version>2.0.1</version>**
</dependency>
on the same program getting
Caused by: java.lang.RuntimeException: java.io.InvalidClassException: org.apache.spark.rpc.netty.RequestMessage; local class incompatible: stream classdesc serialVersionUID = -2221986757032131007, local class serialVersionUID = -5447855329526097695
In Spark-shell on remote
Spark context available as 'sc' (master = local[*], app id = local-1477561433881). Spark session available as 'spark'. Welcome to ____ __ / / _ _____/ / _\\ / _ / _ `/ / ' / / / .__/_, / / / /_\\ version 2.0.1 / /
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101) Type in expressions to have them evaluated. Type :help for more information.
As I am very new to all this, I am not able to figure out the issue in program
I figured it out, posting this in case if someone is going to follow the similar approach.
I had added
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>2.0.0-M3</version>
which comes with scala-library 2.10.6
but there already exists a scala-library 2.11.8 in spark-core
so I had to exclude the earlier one like this
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>2.0.0-M3</version>
<exclusions>
<exclusion>
<artifactId>scala-library</artifactId>
<groupId>org.scala-lang</groupId>
</exclusion>
<exclusion>
<artifactId>scala-reflect</artifactId>
<groupId>org.scala-lang</groupId>
</exclusion>
</exclusions>
</dependency>
Now everything is working fine
This Spark version mismatch:
Update dependency to 2.11.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.