spark + yarn cluster：如何配置物理节点每次仅运行一个执行程序/任务？
[英]spark + yarn cluster: how can i configure physical node to run only one executor\task each time?
问题描述
I have an environment that combines 4 physical nodes with a small amount of RAM and each has 8 CPU cores. 
我的环境将4个物理节点与少量RAM结合在一起，每个节点都有8个CPU内核。 I noticed that spark decides automatically to split the RAM for each CPU. 我注意到spark决定自动为每个CPU分配RAM。 The result is that a memory error occurred. 结果是发生内存错误。 I'm working with big data structures, and I want that each executor will have the entire RAM memory on the physical node (otherwise i'll get a memory error). 我正在使用大数据结构，并且我希望每个执行程序在物理节点上都具有整个RAM内存（否则我会遇到内存错误）。 I tried to configure 'yarn.nodemanager.resource.cpu-vcores 1' on 'yarn-site.xml' file or 'spark.driver.cores 1' on spark-defaults.conf without any success. 我尝试在'yarn-site.xml'文件上配置'yarn.nodemanager.resource.cpu-vcores 1'或在spark-defaults.conf上配置'spark.driver.cores 1'，但没有成功。
1 个解决方案
解决方案1
0 2016-02-23 08:26:41
尝试设置spark.executor.cores 1
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.