Spark RDD接受和删除行

Question

I have an RDD of 1000 elements. I want to take 100 elements from it and then remove those 100 from the initial RDD. But I'm not able to find a way for after trying multiple ways.

var part = dataRDD.take(100)

part is an Array[String]

I want to delete the 100 elements from the 100 of dataRDD.

var dataRDD = dataRDD.filter(row => row != part)

The above doesn't show any error but doesn't remove any rows. dataRDD still has the same 1000 rows.

Can you please guide on how to get this work.

Answer 1

Method "zipWithIndex" can be used for split rdd:

val zipped = rdd.zipWithIndex()
val first100 = zipped.filter(_._2 < 100).keys
val remaining = zipped.filter(_._2 >= 100).keys

Answer 2

You can write like:

var part = sc.parallelize(dataRDD.take(100))
val result = dataRDD.subtract(part)
result.foreach(x => foreach(x))

printing RDD for testing purpose only.

Answer 3

Most probably your code doesn't work as you expect because row != part is always true . != in this context is a references comparison between arrays from java.