如何使用Scala编译spark-cassandra程序？

Question

Lately I started learning spark and cassandra, I know that we can use spark in both python and scala and java, and I 've read docs on this website: https://github.com/datastax/spark-cassandra-connector/blob/master/doc/0_quick_start.md , the thing is, after I create a program named testfile.scala with those codes the document says,(I don't know if I am right using .scala), however, i don't know how to compile it,can anyone guide me what to do with it? Here are the testfile.scala:

import com.datastax.spark.connector._
import com.datastax.spark.connector.streaming._


val conf = new SparkConf(true).set("spark.cassandra.connection.host", "127.0.0.1")

val sc = new SparkContext("spark://127.0.0.1:7077", "test", conf)

val ssc = new StreamingContext(conf, Seconds(n))

val stream = ssc.actorStream[String](Props[SimpleStreamingActor], actorName,          StorageLevel.MEMORY_AND_DISK)

val wc = stream.flatMap(_.split("\\s+")).map(x => (x, 1)).reduceByKey(_ + _).saveToCassandra("streaming_test", "words", SomeColumns("word", "count"))

val rdd = sc.cassandraTable("test", "kv")

println(rdd.count)

println(rdd.first)

println(rdd.map(_.getInt("value")).sum)

Answer 1

Scala projects are compiled by scalac, but it's quite low level: you have to setup build paths and manage all dependencies, so most people fall back to some build tool such as sbt which will manage a lot of stuff for you. The other two commonly used built tools are maven , which is favored by java old-schoolers and gradle , which is more down to earth

> how to import spark-cassandra-connector

I've set up example project . Basically, you define all of your dependencies in built.sbt or it's analog, here is how dependency on spark-cassandra-connector is defined (line #12).

> And, is it a rule that we have to code with class or object

Yes and no. If you code with sbt, all your code files has to be wrapped into object, but, sbt allows you to code in it's shell and code that you input to it is not required to be wrapped (same rules as with ordinary scala REPL). Next, both IDEA and Eclipse have worksheet capabilities , so you can create test.sc and draft your code there.