[英]Spark executor cores not shown in yarn resource manager
Yarn resource manager is not showing the total cores for the spark application. Yarn 资源管理器没有显示 spark 应用程序的总内核数。 For example if we are submiting a spark job with 300 executors and executor-cores is 3. So ideally spark having 900 cores but in yarn resource manager only showing 300 cores.例如,如果我们提交一个有 300 个执行程序的 spark 作业,executor-cores 是 3。那么理想情况下 spark 有 900 个内核,但在 yarn 资源管理器中只显示 300 个内核。
So is this just a display error or is Yarn not seeing the rest of the 600 cores?那么这只是一个显示错误还是 Yarn 没有看到其余的 600 个内核?
Environment: HDP2.2 Scheduler: capacity-scheduler Spark: 1.4.1环境:HDP2.2 调度器:capacity-scheduler Spark:1.4.1
Set放
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
in capacity-scheduler.xml
在capacity-scheduler.xml
中
YARN is running more containers than allocated cores because by default DefaultResourceCalculator is used. YARN 运行的容器多于分配的核心,因为默认情况下使用DefaultResourceCalculator 。 It considers only memory.它只考虑内存。
public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
}
Use DominantResourceCalculator, It uses both cpu and memory.使用 DominantResourceCalculator,它同时使用 cpu 和内存。
you can read more about DominantResourceCalculator here .您可以在此处阅读有关 DominantResourceCalculator 的更多信息。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.