Spark Standalone、YARN 和本地模式有什么区别？

Question

Spark Standalone:

In this mode I realized that you run your Master and worker nodes on your local machine.

Does that mean you have an instance of YARN running on my local machine? Since when I installed Spark it came with Hadoop and usually YARN also gets shipped with Hadoop as well correct? And in this mode I can essentially simulate a smaller version of a full blown cluster.

Spark Local Mode:

This is the part I am also confused on. To run it in this mode I do val conf = new SparkConf().setMaster("local[2]") .

In this mode, it doesn't use any type of resource manager (like YARN) correct? Like it simply just runs the Spark Job in the number of threads which you provide to "local[2]"\ ?

Answer 1

You are getting confused with Hadoop YARN and Spark.

YARN is a software rewrite that decouples MapReduce's resource management and scheduling capabilities from the data processing component, enabling Hadoop to support more varied processing approaches and a broader array of applications.

With the introduction of YARN, Hadoop has opened to run other applications on the platform.

In short YARN is "Pluggable Data Parallel framework".

Apache Spark

Apache spark is a Batch interactive Streaming Framework. Spark has a "pluggable persistent store". Spark can run with any persistence layer.

For spark to run it needs resources. In standalone mode you start workers and spark master and persistence layer can be any - HDFS, FileSystem, cassandra etc. In YARN mode you are asking YARN-Hadoop cluster to manage the resource allocation and book keeping.

When you use master as local[2] you request Spark to use 2 core's and run the driver and workers in the same JVM. In local mode all spark job related tasks run in the same JVM.

So the only difference between Standalone and local mode is that in Standalone you are defining "containers" for the worker and spark master to run in your machine (so you can have 2 workers and your tasks can be distributed in the JVM of those two workers?) but in local mode you are just running everything in the same JVM in your local machine.

Answer 2

local mode
Think of local mode as executing a program on your laptop using single JVM. It can be java, scala or python program where you have defined & used spark context object, imported spark libraries and processed data residing in your system.

YARN
In reality Spark programs are meant to process data stored across machines. Executors process data stored on these machines. We need a utility to monitor executors and manage resources on these machines( clusters). Hadoop has its own resources manager for this purpose. So when you run spark program on HDFS you can leverage hadoop's resource manger utility ie yarn. Hadoop properties is obtained from 'HADOOP_CONF_DIR' set inside spark-env.sh or bash_profile

Spark Standalone
Spark distribution comes with its own resource manager also. When your program uses spark's resource manager, execution mode is called Standalone. Moreover, Spark allows us to create distributed master-slave architecture, by configuring properties file under $SPARK_HOME/conf directory. By Default it is set as single node cluster just like hadoop's psudo-distribution-mode.

Answer 3

I would like to start with JVMs.

So JVM is something that is installed in your system. It is nothing but an interpreter and it is accompanied with some mandatory libraries. So JVM + libraries = JRE.

Firstly any no of JVMs can be formed in your system. ie There can be many JVMs with their own resources allocated eg memory, cores etecetra. But they all share the interpreter, libraries because there is only one copy of them.

Now, in local mode only one JVM takes care of the processing only one machine is used. You can specify how many cores you need, which will in turn become the cores of that JVM. Since you have n number of cores assigned to local mode JVM. You can do partitioning as well here. Maximum partition you can do to the data is no of cores, otherwise there wont be any advantage.

Now in Standalone mode. You will create not one JVM but many. And the spark's own ResourceManager will handle the processing between these JVMs. Since there are multiple JVMs you can have these JVMs in different machines as well. It is a cluster mode.

In cluster mode we use third party resouce managers like YARN/Mesos. You can do same thing as standalone mode.