无法连接到远程系统上的Spark

Question

I am trying to connect to spark master on a remote system through java app

I am using

<dependency> <!-- Spark dependency -->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.0.1</version>
</dependency>

and code

 {
        SparkSession sparkSession = SparkSession.builder().
                           master("spark://ip:7077")
                          .appName("spark session example")
                          .getOrCreate();
        JavaSparkContext sc = new JavaSparkContext(sparkSession.sparkContext());
    }

Getting

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
    at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:713)
    at org.apache.spark.sql.SparkSession$Builder.master(SparkSession.scala:766)
    at com.mobelisk.spark.JavaSparkPi.main(JavaSparkPi.java:9)

Also If I change to

<dependency> <!-- Spark dependency -->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            **<version>2.0.1</version>**
        </dependency>

on the same program getting

Caused by: java.lang.RuntimeException: java.io.InvalidClassException: org.apache.spark.rpc.netty.RequestMessage; local class incompatible: stream classdesc serialVersionUID = -2221986757032131007, local class serialVersionUID = -5447855329526097695

In Spark-shell on remote

Spark context available as 'sc' (master = local[*], app id = local-1477561433881). Spark session available as 'spark'. Welcome to ____ __ / / _ _____/ / _\\ / _ / _ `/ / ' / / / .__/_, / / / /_\\ version 2.0.1 / /

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101) Type in expressions to have them evaluated. Type :help for more information.

As I am very new to all this, I am not able to figure out the issue in program

Answer 1

I figured it out, posting this in case if someone is going to follow the similar approach.

I had added

<groupId>com.datastax.spark</groupId>
            <artifactId>spark-cassandra-connector_2.10</artifactId>
<version>2.0.0-M3</version>

which comes with scala-library 2.10.6

but there already exists a scala-library 2.11.8 in spark-core

so I had to exclude the earlier one like this

<dependency>
            <groupId>com.datastax.spark</groupId>
            <artifactId>spark-cassandra-connector_2.10</artifactId>
            <version>2.0.0-M3</version>
            <exclusions>
                <exclusion>
                    <artifactId>scala-library</artifactId>
                    <groupId>org.scala-lang</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-reflect</artifactId>
                    <groupId>org.scala-lang</groupId>
                </exclusion>
            </exclusions>
</dependency>

Now everything is working fine

Answer 2

This Spark version mismatch:

• you use 2.10 in project.
• cluster uses 2.11

Update dependency to 2.11.