使用spark cassandra连接器和SBT编译错误

Question

I'm trying to get the DataStax spark cassandra connector working. I've created a new SBT project in IntelliJ, and added a single class. The class and my sbt file is given below. Creating spark contexts seem to work, however, the moment I uncomment the line where I try to create a cassandraTable, I get the following compilation error:

Error:scalac: bad symbolic reference. A signature in CassandraRow.class refers to term catalyst in package org.apache.spark.sql which is not available. It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling CassandraRow.class.

Sbt is kind of new to me, and I would appreciate any help in understanding what this error means (and of course, how to resolve it).

name := "cassySpark1"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.1.0"

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector" % "1.1.0" withSources() withJavadoc()

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector-java" % "1.1.0-alpha2" withSources() withJavadoc()

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

And my class:

import org.apache.spark.{SparkConf, SparkContext}

import com.datastax.spark.connector._

object HelloWorld { def main(args:Array[String]): Unit ={ System.setProperty("spark.cassandra.query.retry.count", "1")

 val conf = new SparkConf(true) .set("spark.cassandra.connection.host", "cassandra-hostname") .set("spark.cassandra.username", "cassandra") .set("spark.cassandra.password", "cassandra") val sc = new SparkContext("local", "testingCassy", conf) 

> //val foo = sc.cassandraTable("keyspace name", "table name")

 val rdd = sc.parallelize(1 to 100) val sum = rdd.reduce(_+_) println(sum) } } 

Answer 1

您需要将spark-sql添加到依赖项列表中

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.0"

Answer 2

Add library dependency in your project's pom.xml file. It seems they have changed the Vector.class dependencies location in the new refactoring.