如何使用scala在apache spark中存储操作的结果

Question

How to store the result generated from an action like: count in an output directory, in apache Spark Scala?

    val countval= data.map((_,"")).reduceByKey((_+_)).count

The below command does not work as count is not stored as RDD:

    countval.saveAsTextFile("OUTPUT LOCATION")

Is there any way to store countval into local/hdfs location?

Answer 1

After you call count it is no longer RDD.

Count is just Long and it does not have saveAsTextFile method.

If you want to store your countval you have to do it like with any other long, string, int...

Answer 2

what @szefuf said is correct, after count you have a Long which you can save any way you want. If you want to save it as an RDD with .saveAsTextFile() you have to convert it to an RDD:

 sc.parallelize(Seq(countval)).saveAsTextFile("/file/location")

The parallelize method in SparkContext turns a collection of values into an RDD, so you need to turn the single value to a single-element sequence first. Then you can save it.