使用Scala TTL選項將火花節省到Cassandra

Question

我在Scala中使用Cassandra Spark連接器。 這是我的示例代碼，用於將數據保存到Cassandra。

val data = rdd.map
(f => new CassandraRow(IndexedSeq("pk", "count"), IndexedSeq(f._1, f._2.toString())))

data.saveToCassandra("keyspace", "table")

然后，我嘗試使用TTL。 這是我添加TTL的示例代碼。

添加導入

import com.datastax.spark.connector.writer.{TTLOption, WriteConf}

並將TTL添加到saveToCassandra

data.saveToCassandra
("keyspace", "table", writeConf = WriteConf(ttl = TTLOption.constant(604800))

但是，當我編譯它時，它會出現錯誤。

錯誤訊息

[error] bad symbolic reference. A signature in TTLOption.class refers to term streaming
[error] in package org.apache.spark which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling TTLOption.class.
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 4 s, completed Mar 31, 2015 11:55:14 AM

build.sbt

name := "Cassandra"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.1"

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.2.1"

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.0-rc2"

我的代碼有問題嗎？

Answer 1

這是當前連接器中的一個已知錯誤，與對流Spark庫的依賴性有關。

https://datastax-oss.atlassian.net/browse/SPARKC-113

如果這阻塞了，並且您需要一種解決方法，則可以僅將spark-streaming庫作為依賴項包含在構建文件中。

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.2.1"