如何向 Spark 中的现有 RDD 添加更多 RDD？

Question

I have a RDD and want to add more RDD to it. How can I do it in Spark? I have code like below. I want to return RDD from the dStream I have.

JavaDStream<Object> newDStream = dStream.map(this);
JavaRDD<Object> rdd = context.sparkContext().emptyRDD();
return newDStream.wrapRDD(context.sparkContext().emptyRDD());

I do not find much documentation about wrapRDD method of JavaDStream class provided by Apache Spark.

Answer 1

Since RDD is immutable , what you can do is use sparkContext.parallelize to create a new RDD and return the new one.

List<Object> objectList = new ArrayList<Object>;
objectList.add("your content");

JavaRDD<Object> objectRDD = sparkContext.parallelize(objectList);
JavaRDD<Object> newRDD = oldRDD.union(objectRDD);

See https://spark.apache.org/docs/latest/rdd-programming-guide.html#parallelized-collections

Answer 2

You can use JavaStreamingContext.queueStream and fill it with a Queue<RDD<YourType>> :

public JavaInputDStream<Object> FillDStream() {
    LinkedList<RDD<Object>> rdds = new LinkedList<RDD<Object>>();
    rdds.add(context.sparkContext.emptyRDD());
    rdds.add(context.sparkContext.emptyRDD());

    JavaInputDStream<Object> filledDStream = context.queueStream(rdds);
    return filledStream;
}