使用 Spark 流式传输 cassandra 中的最新数据

Question

I continuously have data being written to cassandra from an outside source.

Now, I am using spark streaming to continuously read this data from cassandra with the following code:

val ssc = new StreamingContext(sc, Seconds(5))

val cassandraRDD = ssc.cassandraTable("keyspace2", "feeds")


val dstream = new ConstantInputDStream(ssc, cassandraRDD)


dstream.foreachRDD { rdd =>
 println("\n"+rdd.count())
}

ssc.start()
ssc.awaitTermination()
sc.stop()

However, the following line:

val cassandraRDD = ssc.cassandraTable("keyspace2", "feeds")

takes the entire table data from cassandra every time. Now just the newest data saved into the table.

What I want to do is have spark streaming read only the latest data, ie, the data added after its previous read.

How can I achieve this? I tried to Google this but got very little documentation regarding this.

I am using spark 1.4.1 , scala 2.10.4 and cassandra 2.1.12 .

Thanks!

EDIT:

The suggested duplicate question (asked by me) is NOT a duplicate, because it talks about connecting spark streaming and cassandra and this question is about streaming only the latest data. BTW, streaming from cassandra IS possible by using the code I provided. However, it takes the entire table every time and not just the latest data.

Answer 1

Cassandra 将有一些低级工作，允许将传入 Cassandra 的新突变通知外部系统（索引器、Spark 流等），请阅读： https : //issues.apache.org/jira/browse/卡桑德拉-8844