Kafka Spark流式缓存

Question

I have been getting data from one kafka topics in the form of JavaPairInputDStream (with twitter streaming api), plan is getting data from two topics checking for duplication with tweet_id and if its not in the package (package is for sending back to kafka), add it. Also i want to cache data for x mins then work on it.

I can get data from kafka topic and output it with

stream.foreachRDD(rdd -> {
    System.out.println("--- New RDD with " + rdd.partitions().size()
     + " partitions and " + rdd.count() + " records");
     rdd.foreach(record -> System.out.println(record._2));});

But i cant manage to cache it. Tried rdd.cache() and persist with count() . but it doesn't seem to do trick or i just wasn't able to understand it.

Anyone can guide me how to do this stuff?

Answer 1

Okay so its impossible to cache rdd like this it seems. I created another rdd and I'm using union() whenever stream creates new rdd and caching this way.