無法使用Spark Cassandra連接器1.5.0連接Cassandra 3.0

Question

問題 -無法使用Spark Cassandra連接器1.5.0連接Cassandra 3.0

背景 -我嘗試通過使用提供的Spark cassandra連接器1.5.0從Spark 1.5.0連接使用Cassandra 3.0，但出現以下錯誤-

根據DataStax Spark Cassandra Connector 文檔 ，它說Spark連接器1.5可以用於Spark 1.5.0 / 1.6.0中的Cassandra 3.0。

您能否建議我在這里缺少任何步驟？

嘗試的方法

1. 我試圖從Spark流和核心jar中排除Guava依賴項

2. 在“ pom.xml”中添加了單獨的Guava依賴項

提前致謝。

---

16/04/26 09:45:07 WARN TaskSetManager: Lost task 4.0 in stage 1.0 (TID 16, ip-172-31-23-23.ec2.internal): java.lang.ExceptionInInitializerError
    at  com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFactory.scala:35)
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:87)
    at 
---
--
Caused by: java.lang.IllegalStateException: Detected Guava issue #1635 which indicates that a version of Guava less than 16.01 is in use.  This introduces codec resolution issues and potentially other incompatibility issues in the driver.  Please upgrade to Guava 16.01 or later.
    at com.datastax.driver.core.SanityChecks.checkGuava(SanityChecks.java:62)
    at com.datastax.driver.core.SanityChecks.check(SanityChecks.java:36)
    at com.datastax.driver.core.Cluster.<clinit>(Cluster.java:67)
    ... 23 more16/04/26 09:45:07 WARN TaskSetManager: Lost task 4.0 in stage 1.0 (TID 16, ip-172-31-23-23.ec2.internal): java.lang.ExceptionInInitializerError
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFactory.scala:35)
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:87)
    at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:153)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
    at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)

---

Answer 1

連接器隨附的Guava版本與Spark隨附的版本之間存在已知沖突。 您如何嘗試對Guava庫進行着色？

嘗試將其添加到您的build.sbt文件中：

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shadeio.@1").inAll
)