无法使用Spark Cassandra连接器1.5.0连接Cassandra 3.0

Question

Issue - Unable to connect Cassandra 3.0 using Spark Cassandra connector 1.5.0

Background - I tried to connect Cassandra 3.0 using from Spark 1.5.0 by using provided Spark cassandra connector 1.5.0 but I am getting below error -

As per the DataStax Spark Cassandra Connector document , it says that Spark connector 1.5 can be used to Cassandra 3.0 from Spark 1.5.0/1.6.0.

Could you please suggest me is I am missing any step here?

Tried approach

1. I tried to exclude Guava dependency from Spark streaming and core jar

2. Added the separate Guava dependency in "pom.xml"

Thanks in advance.

---

16/04/26 09:45:07 WARN TaskSetManager: Lost task 4.0 in stage 1.0 (TID 16, ip-172-31-23-23.ec2.internal): java.lang.ExceptionInInitializerError
    at  com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFactory.scala:35)
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:87)
    at 
---
--
Caused by: java.lang.IllegalStateException: Detected Guava issue #1635 which indicates that a version of Guava less than 16.01 is in use.  This introduces codec resolution issues and potentially other incompatibility issues in the driver.  Please upgrade to Guava 16.01 or later.
    at com.datastax.driver.core.SanityChecks.checkGuava(SanityChecks.java:62)
    at com.datastax.driver.core.SanityChecks.check(SanityChecks.java:36)
    at com.datastax.driver.core.Cluster.<clinit>(Cluster.java:67)
    ... 23 more16/04/26 09:45:07 WARN TaskSetManager: Lost task 4.0 in stage 1.0 (TID 16, ip-172-31-23-23.ec2.internal): java.lang.ExceptionInInitializerError
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFactory.scala:35)
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:87)
    at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:153)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
    at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)

---

Answer 1

There is a known conflict between the Guava version shipped with the connector, and the one shipped with spark. How did you try to shade the Guava library?

Try to add this to your build.sbt file:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shadeio.@1").inAll
)