如果执行器失败，Spark with External Shuffle Service 可以使用保存的 shuffle 文件吗？

Question

The Spark docs state:

a Spark executor exits either on failure or when the associated application has also exited. In both scenarios, all state associated with the executor is no longer needed and can be safely discarded.

However, in the scenario where the Spark cluster configuration and dataset are such that occasional executor failure OOM occurs deep into a job, it is far more preferable for the shuffle files written by the dead executor to continue to be available to the job rather than have them recomputed.

In such a scenario with the External Shuffle Service enabled, I have appeared to observe Spark continuing to fetch the afore mentioned shuffle files and only rerunning the tasks that were active at the time when the executor died. In contrast, with the External Shuffle Service disabled I have seen Spark rerun a proportion of previously completed stages to recompute lost shuffle files as expected.

So can Spark with the External Shuffle Service enabled use saved shuffle files in the event of Executor failure as I have appeared to observe? I think so, but the documentation makes me doubt it.

I am running Spark 3.0.1 with Yarn on EMR 6.2 with dynamic allocation disabled.

Also, pre-empting comments, of course it is preferable to configure the cluster so that executor OOM never occurs. However, when initially aiming to complete an expensive Spark job the optimal cluster configuration is not yet achieved. It is at this time that shuffle reuse in the face of executor failure is valuable.

Answer 1

The sentence you quoted:

a Spark executor exits either on failure or when the associated application has also exited. In both scenarios, all state associated with the executor is no longer needed and can be safely discarded.

is from the "Graceful Decommission of Executors" section.

That feature main intention is to provide a solution when Kubernetes is used as a resource manager. Where external shuffle service is not available. It is migrating the disk persisted RDD blocks and shuffle blocks to the remaining executors.

In case of Yarn when external shuffle service is enabled the blocks will be fetched from the external shuffle service which is running as an auxiliary service of the Yarn (within the node manager). That service knows the executors internal directory structure and able to serve the blocks (as it is on the same host). This way when the node survives and just the executor dies the blocks won't be lost.

Answer 2

Dynamic resource allocation requires ESS to be enabled in Yarn. See this in the same doc.

ESS is an independent service running in node manager. When executor dies, ESS on the same node is able to serve its map output files when remote executors try to fetch.

Answer 3

I think the Spark Blacklist Design Doc answers the question (referenced in the HealthTracker class comments):

Spark already does some handling of failed nodes when it fetches a shuffle block. This automatically leads to spark marking the shuffle data as missing, and using the stage failure mechanism to regenerate the missing blocks based on lineage.

ie It is fetch failure that triggers stage failure and subsequent shuffle block regeneration. Fetch failure and hence subsequent stage failure will not occur on executor death when the Shuffle Service is enabled and serving the fetch requests. Furthermore, there is nothing in either the blacklist design, or the driver's HeartBeatReceiver , to explicitly remove shuffle blocks written by a lost executor. Hence, shuffle blocks written by a dead executor will continue to be served by the shuffle service on the associated node.