DSE Cassandra Spark Error

Question

I spent the whole day trying to figure out a way to resolve this but no luck.

Each time my java application tries to connect to my DSE Spark master, It throws an error:

Association with remote system [akka.tcp://sparkMaster@192.168.2.104:7077] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].

Tailing cassandra log, I can see an incompatibilty in the class version:

ERROR [sparkMaster-akka.actor.default-dispatcher-2] 2015-06-09 16:39:12,066  
Slf4jLogger.scala:66 - org.apache.spark.deploy.ApplicationDescription; local 
class incompatible: stream classdesc serialVersionUID = 7674242335164700840, 
local
class serialVersionUID = 2812534333379744973
java.io.InvalidClassException: 
org.apache.spark.deploy.ApplicationDescription; local class incompatible: 
stream classdesc serialVersionUID = 7674242335164700840, local class 
serialVersionUID = 2812534333379744973

Any idea why this is happening ?

It's a version mismatch but this is not clear as what to do to resolve it.

Am running DSE 4.7 and my POM file is like below:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.10</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector-java_2.10</artifactId>
    <version>1.2.1</version>
</dependency>

<dependency>
    <groupId>com.datastax</groupId>
    <artifactId>dse</artifactId>
    <version>version number</version>
    <scope>system</scope>
    <systemPath>${project.basedir}/lib/dse-4.7.0.jar</systemPath>
</dependency>

Question is which version of cassandra connector works with DSE 4.7 ?

Answer 1

There is a version mismatch. DSE 4.7 ships with Spark 1.2.1 but you are building your jar against 1.1.1.