使用Scala TTL选项将火花节省到Cassandra

Question

我在Scala中使用Cassandra Spark连接器。 这是我的示例代码，用于将数据保存到Cassandra。

val data = rdd.map
(f => new CassandraRow(IndexedSeq("pk", "count"), IndexedSeq(f._1, f._2.toString())))

data.saveToCassandra("keyspace", "table")

然后，我尝试使用TTL。 这是我添加TTL的示例代码。

添加导入

import com.datastax.spark.connector.writer.{TTLOption, WriteConf}

并将TTL添加到saveToCassandra

data.saveToCassandra
("keyspace", "table", writeConf = WriteConf(ttl = TTLOption.constant(604800))

但是，当我编译它时，它会出现错误。

错误讯息

[error] bad symbolic reference. A signature in TTLOption.class refers to term streaming
[error] in package org.apache.spark which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling TTLOption.class.
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 4 s, completed Mar 31, 2015 11:55:14 AM

build.sbt

name := "Cassandra"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.1"

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.2.1"

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.0-rc2"

我的代码有问题吗？

Answer 1

这是当前连接器中的一个已知错误，与对流Spark库的依赖性有关。

https://datastax-oss.atlassian.net/browse/SPARKC-113

如果这阻塞了，并且您需要一种解决方法，则可以仅将spark-streaming库作为依赖项包含在构建文件中。

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.2.1"