[英]Deploy mode in “SPARK-SUBMIT”
In SPARK-SUBMIT , what is the difference between "yarn" , "yarn-cluster" , "yarn-client" deploy modes ? 在SPARK-SUBMIT中,“纱线”,“纱线簇”,“纱线 - 客户”部署模式之间有什么区别?
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn-cluster \ # can also be `yarn-client` for client mode
--executor-memory 20G \
--num-executors 50 \
/path/to/examples.jar \
1000
https://spark.apache.org/docs/1.1.0/submitting-applications.html https://spark.apache.org/docs/1.1.0/submitting-applications.html
For Spark on YARN, you can specify either yarn-client or yarn-cluster. 对于YARN上的Spark,您可以指定yarn-client或yarn-cluster。 Yarn-client runs driver program in the same JVM as spark submit, while yarn-cluster runs Spark driver in one of NodeManager's container.
Yarn-client在与spark提交相同的JVM中运行驱动程序,而yarn-cluster在NodeManager的一个容器中运行Spark驱动程序。
From the documentation: https://spark.apache.org/docs/1.1.0/running-on-yarn.html There are two deploy modes that can be used to launch Spark applications on YARN. 从文档: https : //spark.apache.org/docs/1.1.0/running-on-yarn.html有两种部署模式可用于在YARN上启动Spark应用程序。 In yarn-cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application.
在yarn-cluster模式下,Spark驱动程序在应用程序主进程内运行,该进程由群集上的YARN管理,客户端可以在启动应用程序后消失。 In yarn-client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.
在yarn-client模式下,驱动程序在客户端进程中运行,而应用程序主服务器仅用于从YARN请求资源。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.