[英]How to change the number of CPUs each worker uses in Spark Standalone mode?
I could run master and slave nodes in Spark Standalone mode on the machine with OS Windows 7, 8 cores CPU and 16 GB RAM. 我可以在具有OS Windows 7,8核CPU和16 GB RAM的机器上以Spark Standalone模式运行主节点和从节点。
Now I'm trying to change the number of workers on this machine. 现在我正在尝试改变这台机器上的工人数量。
As I read on Spark's site, launching scripts are not currently supported in Windows OS. 正如我在Spark网站上看到的那样,Windows操作系统目前不支持启动脚本。 So I start the master and workers manually. 所以我手动启动主人和工人。
In driver program I define: 在驱动程序中我定义:
SparkConf conf = new SparkConf().setAppName("myapplication")
.setMaster("spark://fujitsu11:7077")
.setJars(new String[] {"target/maven1-0.0.1-SNAPSHOT-driver.jar"})
.set("spark.home","D:/spark")
.set("spark.executor.memory", "2g")
.set("spark.worker.instances","2")
.set("spark.worker.cores", "4"); // I allocate four cores for each worker
JavaSparkContext sc = new JavaSparkContext(conf);
Then in Cygwin I start by hand master daemon and two workers. 然后在Cygwin,我开始手动掌握守护进程和两名工人。
But when I run my application and follow on http://localhost:8080 I see, that both workers trying to use all 8 cores... 但是,当我运行我的应用程序并按照http:// localhost:8080时,我看到,两个工作人员都试图使用所有8个核心......
How could I change the number of cores the instances use? 我怎样才能更改实例使用的核心数量?
The number of the CPU/cores a worker uses is given as an environment variable at start time, and you could set it in two places: 工作人员使用的CPU /核心数在开始时作为环境变量给出,您可以在两个位置设置它:
conf/spark-env.sh
as SPARK_WORKER_CORES
. 在conf/spark-env.sh
为SPARK_WORKER_CORES
。 --cores 4
. 作为--cores 4
启动工作器守护程序时的命令行参数。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.