非 rdd spark scala 的 saveAsTextFile

Question

I have this code :

def contentSizeStats(rdd: RDD[ApacheAccessLog]) = {
  val contentSizes = rdd.map(x=>x.contentSize).cache()
  val count = contentSizes.count()
   if (count == 0) { null }
   (count, contentSizes.reduce(_+_), contentSizes.min(), contentSizes.max())
}

The return value of this function is (Long, String, String, String), I am trying to save the result of this function to a text file using saveAsTextFile and I cant since the result is not an RDD,any idea how to implement this ?

Answer 1

You can convert the return value to an RDD and then call saveAsTextFile on it.

val outputRdd = sc.parallelize(outputTuple.productIterator.toArray)
outputRdd.saveAsTextFile(outputDirectory)