Spark在Gzip中编码并发送到S3 - java.io.IOException：设备上没有剩余空间

Question

I'm trying to GZIP and send an RDD over to S3 like so:

dwPartitioned.saveAsTextFile(s"s3n://$accessKey:$secretKey@bucket", classOf[GzipCodec])

The job starts running and shortly after comes up with:

org.apache.spark.SparkException: Job aborted due to stage failure:  ... : java.io.IOException: No space left on device

I read that because of the encoding there is some shuffling done which requires temporary files to be generated. Is that true? Am I misusing the functionality? Is there something that I can optimize here?

More importantly - how can I achieve this in memory?

If you need more info I'll gladly append it.

Answer 1

By default, spark uses " /tmp " to save intermediate files. When the job is running, you can tab " df -h " to see the used space of fs mounted at "/" growing up. When the space of the dev is runned out of, this exception is throwed. To solve the problem, set the SPARK_LOCAL_DIRS in the SPARK_HOME/conf/spark_defaults.conf with a path in a fs leaving enough space.