Spark cache/persist vs shuffle files

Question

I am trying to understand the benefit of spark caching/persist mechanism

AFAIK, spark always persists the RDDs after a shuffle as an optimizaiton. So what benefit does cache/persist call provide? I am assuming the cache/persist happens in memory so the only benefit is that it won't read from the disk, correct?

Answer 1

Two differences come to mind

1. During shuffle, intermediate data (data that need to be shuffled across nodes) gets saved so as to avoid reshuffling. This gets reflected in Spark UI as skipped stages. With cache/persist, you are caching the processed data .
2. You are in control of what need to be cached but you doesn't have explicit control on caching shuffled data (it is behind the scenes optimization).

Answer 2

what benefit does cache/persist call provide?

one example that comes to me at once is that the spark reading process, if you read some data from file system and you do two separate sets of transformations(and finally action) on it, you will load two times the source data(you can check your UI and you will see two loads), but if you cache it, the load process will only happen once.

cache/persist happens in memory so the only benefit is that it won't read from the disk

nope, cache and persist happen in different levels and memory is the default level: check out here: https://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence

Answer 3

Spark dataframes are lazy evaluated and if you do something like

val a = df1.join(df2)
val b = a.groupBy(col).agg(...)
a.write.parquet(...)
b.write.parquet(...)

then df1 and df2 will be scanned and joined twice, once in each write operation.

That is, unless you cache or persist them.

Persistence for shuffle is a different thing altogether, and deals with the internals of the shuffle operation - not something that impacts what you see at the application layer.