Spark - StorageLevel (DISK_ONLY vs MEMORY_AND_DISK) 和内存不足的 Java 堆空间

Question

Lately I've been running a memory-heavy spark job and started to wonder about storage levels of spark. I persisted one of my RDDs as it was used twice using StorageLevel.MEMORY_AND_DISK . I was getting OOM Java heap space during the job. Then, when I removed the persist completely, the job has managed to go through and finish.

I always thought that the MEMORY_AND_DISK is basically a fully safe option - if you run out of memory, it spills the object to disk, done. But now it seemed that it did not really work in the way I expected it to.

This derives two questions:

1. If MEMORY_AND_DISK spills the objects to disk when executor goes out of memory, does it ever make sense to use DISK_ONLY mode (except some very specific configurations like spark.memory.storageFraction=0 )?
2. If MEMORY_AND_DISK spills the objects to disk when executor goes out of memory, how could I fix the problem with OOM by removing the caching? Did I miss something and the problem was actually elsewhere?

Answer 1

So, after few years ;) that's what I believe happened:

• Caching is not a way to save execution memory. The best you can do is not to lose execution memory ( DISK_ONLY ) when caching.
• It's most likely the lack of execution memory that caused my job to throw OOM error, although I don't remember the actual use case.
• I used MEMORY_AND_DISK caching and the MEMORY part took its part from the unified region which made it impossible for my job to finish (since the Execution = Unified - Storage memory was not enough to perform the job)
• Due to above, when I removed caching at all, it took slower, but the job had enough execution memory to finish.
• With DISK_ONLY caching it seems that the job would therefore finish as well (although not necessarily faster).

https://spark.apache.org/docs/latest/tuning.html#memory-management-overview

Answer 2

MEMORY_AND_DISK doesn't "spill the objects to disk when executor goes out of memory". It tells Spark to write partitions not fitting in memory to Disk so they will be loaded from there when needed.

Dealing with huge datasets you should definately consider persisting data to DISK_ONLY. https://spark.apache.org/docs/latest/rdd-programming-guide.html#which-storage-level-to-choose