Spark saving to Cassandra with Scala TTL Option

Question

I use Cassandra Spark connector in Scala. Here is my sample code that is working to save data to Cassandra.

val data = rdd.map
(f => new CassandraRow(IndexedSeq("pk", "count"), IndexedSeq(f._1, f._2.toString())))

data.saveToCassandra("keyspace", "table")

Then, I try to use TTL. Here is my sample code for adding TTL.

adding import

import com.datastax.spark.connector.writer.{TTLOption, WriteConf}

and adding TTL to saveToCassandra

data.saveToCassandra
("keyspace", "table", writeConf = WriteConf(ttl = TTLOption.constant(604800))

However, when I compiled it, it gets Errors.

Error Messages

[error] bad symbolic reference. A signature in TTLOption.class refers to term streaming
[error] in package org.apache.spark which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling TTLOption.class.
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 4 s, completed Mar 31, 2015 11:55:14 AM

build.sbt

name := "Cassandra"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.1"

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.2.1"

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.0-rc2"

Is there problem with my code??

Answer 1

This is a known bug in the current connector relating to a dependency on the streaming spark libraries.

https://datastax-oss.atlassian.net/browse/SPARKC-113

If this is blocking and you need a workaround you can just include the spark-streaming library as a dependency in the build file.

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.2.1"