why Spark executor needs to connect with Worker

Question

When I kicked off one Spark job I will find the Executor startup command line as following:

bin/java -cp /opt/conf/:/opt/jars/* -Xmx1024M -Dspark.driver.port=56559 
org.apache.spark.executor.CoarseGrainedExecutorBackend 
--driver-url spark://CoarseGrainedScheduler@10.1.140.2:56559 
--executor-id 1 --hostname 10.1.140.5 --cores 2 
--app-id app-20161221132517-0000 
--worker-url spark://Worker@10.1.140.5:56451

From above command we would find the line --worker-url spark://Worker@10.1.140.5:56451 ,that's I'm curious about, why Executor needs to communicate with Worker, in my mind executor only needs to talk with other executors and Driver.

Answer 1

You can see in the above image that Executors are part of worker nodes .

Application : User program built on Spark. Consists of a driver program and executors on the cluster.

Worker node : Any node that can run application code in the cluster

Executor : A process launched for an application on a worker node , that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.

Source

Answer 2

Executor fate is connected with the worker fate. If worker is abnormally terminated executors have to be able to detect this fact and stop themselves. Without this process one could end up with "ghost" executors.