Scala Spark RDD：rdd.take（）仅接受整数

Question

val fileContent=sc.textFile(path)
val x=fileContent.count()/2
fileContent.take(x) // gives error x: should be integer

x is long. I can do `x.toInt , but what if x is too large to converted to Int?

How to get second half of the rdd?

Answer 1

If you want all the elements you can use the collect method on the RDD.

If you want specifically the first half of it which is already more than Integers max size, you could as others suggested filter out the half you dont need, so basically transforming it to another rdd with less items and collect on that. Like this:

  val sizeOfRdd = fileContent.count()
  fileContent
    .zipWithIndex // assign an index to every element
    .filter(_._2 < sizeOfRdd/2) // filter out the first half
    .map(_._1) // drop the index
    .collect() // take every element 

Note , both take and collect will move the elements in the dataset to the driver, where you might run into memory issues in case you have a lot of elements in the RDD (which I assume you have).

Answer 2

可以使用randomSplit（）将RDD分成较小的RDD数组，然后可以针对每个RDD在循环中执行take（）。