如何将对象文件另存为Spark中的其他目录？

Question

I hava an RDD below:

Array(
(0, "xx"),
(1, "xx"),
(2, "xx"),
(1, "yy")
)

I want to save it to different directories by the key. For example, to create 3 files in those directories:

0/part-00000 // xx
1/part-00000 // xx and yy
2/part-00000 // xx

Through saveAsHadoopFile and MultipleTextOutputFormat , I can do it in text format . However, this RDD contains huge complex data. Saving it in compressed format may be better, like what saveAsObjectFile does.

MultipleSequenceFileOutputFormat may help me realize it, but how to use it correctly?

---

EDIT :

I have tried this to do it in text format:

.saveAsHadoopFile(outputPath, classOf[Any], classOf[Any], classOf[MultiOutputFormat])

  class MultiOutputFormat() extends MultipleTextOutputFormat[Any, Any] {

    override def generateActualKey(key: Any, value: Any): Any = {
      NullWritable.get()
    }

    override def generateFileNameForKeyValue(key: Any, value: Any, name: String): String = {
      key.asInstanceOf[Int] + "/" + super.generateFileNameForKeyValue(key, value, name)
    }
  }

Answer 1

What version of spark are you using?

Have you tried something like

.repartition(3).saveAsTextFile("/path/to/output", classOf[GzipCodec])

or

sc.hadoopConfiguration.setClass(FileOutputFormat.COMPRESS_CODEC, classOf[GzipCodec], classOf[CompressionCodec])

?