怎么把RDD [CassandraRow]转换成DataFrame？

Question

Currently this is how I am transforming Cassandrarow RDD to dataframe:

val ssc = new StreamingContext(sc, Seconds(15))

val dstream = new ConstantInputDStream(ssc, ssc.cassandraTable("db", "table").select("createdon"))

import sqlContext.implicits._

dstream.foreachRDD{ rdd =>
    val dataframeJobs = rdd.map(w => (w.dataAsString)).map(_.split(":")).map(x =>(x(1))).map(_.split(" ")).map(x =>(x(1))).toDF("ondate")
}

As you can see, I am converting first cassandraRow rdd to string first, and then mapping to the format I want. I find this method to get complicated as when the rdd contains multiple coloumns instead of just one (createdon) as shown in the example.

Is there any other alternative and easy way to convert cassandraRow RDD to dataframe?

My build.sbt is as follows:

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.1",
  "org.apache.spark" %% "spark-core" % "2.0.2" % "provided",
  "org.apache.spark" %% "spark-sql" % "2.0.2",
  "org.apache.spark" %% "spark-streaming" % "2.0.2"
)

Answer 1

I figured out an alternative way which could work with any number of coloumns effectively:

rdd.keyBy(row => (row.getString("createdon"))).map(x => x._1).toDF("ondate")

Answer 2

Quoting the scaladoc of SparkContextFunctions (removing the implicit params):

cassandraTable[T](keyspace: String, table: String): CassandraTableScanRDD[T] Returns a view of a Cassandra table as CassandraRDD. This method is made available on SparkContext by importing com.datastax.spark.connector._

Depending on the type parameter passed to cassandraTable, every row is converted to one of the following:

• an CassandraRow object (default, if no type given)
• a tuple containing column values in the same order as columns selected by CassandraRDD#select
• object of a user defined class, populated by appropriate ColumnMapper

So, I'd recommend using the following:

ssc.cassandraTable[String]("db", "table").select("createdon")

That should give you the easiest possible way to access createdon per the docs.

---

I'm also wondering why you don't use DataFrame that spark-cassandra-connector supports as described in Datasets . With that your code might get slightly simpler.

You could try to replace Spark Streaming ( almost officially obsolete) with Spark SQL's Structured Streaming :

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. You can express your streaming computation the same way you would express a batch computation on static data. The Spark SQL engine will take care of running it incrementally and continuously and updating the final result as streaming data continues to arrive.

I'm not sure however if Cassandra Spark Connector supports it.