如何将RDD中的一系列元素复制到较小的RDD中

Question

I have the following RDD[String]:

val rdd = sc.makeRDD(Seq("paul", "jim,", "joe", "mary", "sean", "peter", "lucy")) 

What I would like to be able to do is to generate a smallerRDD by being able to copy a range of lines from the above master rdd into a smaller rdd.

Use case: When spinning through RDDs in spark unusual situations can arise, more often than not certain lines/records in RDDs can cause problems.

Being able to programatically copy one to the other use a usefull feature indeed as I could not find a canned rdd method to do this. see my solution below.

Answer 1

val rdd = sc.makeRDD(Seq("paul", "jim", "joe", "mary", "sean", "peter", "lucy")) 

val startIndex = 1
val endIndex = 5
val shortRdd=rdd.zipWithIndex().filter { case (_, idx) => idx >= startIndex && idx <= endIndex }.map(p=>p._1)
shortRdd.count
shortRdd.foreach(println)

Step1: Lets see whats inside the RDD:

rdd.foreach(println)
peter
lucy
jim
joe
paul
mary
sean

Step2: Apply a transformation to append index, notice the an index value is now applied to each line.

rdd.zipWithIndex().foreach(println)
(peter,5)
(jim,1)
(joe,2)
(paul,0)
(mary,3)
(sean,4)
(lucy,6)

Step3: Apply filters on the index position, pull indexes between start and end index position

rdd.zipWithIndex().filter { case (_, idx) => idx >= startIndex && idx <= endIndex }.foreach(println)
(mary,3)
(sean,4)
(jim,1)
(peter,5)
(joe,2)

Step4: Map back to single element in each line

rdd.zipWithIndex().filter { case (_, idx) => idx >= startIndex && idx <= endIndex }.map(p=>p._1).foreach(println)
mary
jim
joe
peter
sean

I performed this process on RDD with lines of 100k or more without any issues. Let me know how this performs with larger RDD.

Thats it! Paul.