简体   繁体   中英

How to understand spark-submit script master is YARN?

We have all 6 machine, hdfs and yarn service on all node, 1 master and 6 slaves. And we install Spark on 3 machine, 1 master, 3 workers ( 1 node master + worker) . We know when --master spark://[host]:[port] , the job will run only 3 node use standalone mode. And when use spark-submit --master yarn submit a jar, it's would use all 6 server cpu and memory or just use 3 spark worker node machine ? And if can run all 6 node, How left 3 server can know it's the Spark job?

Spark: 2.3.1 Hadoop: 2.7.3

在纱线模式下,火花提交将资源分配资源发送到纱线,并且容器将基于资源可用性在不同的节点管理器上启动。

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM