Will YARN working on NUMA respect node memory locality?
Question
I'm working with the Giraph-based application that makes heavy use of memory in a NUMA system. It frequently writes and reads to the memory and has multiple threads. Assuming I schedule 4 workers with as many cores as there are cores per chip would that mean they would be pinned to specific CPU so that they can only use local NUMA memory?
1 answers
solution1
1 2021-02-11 13:49:33
It depends on what version of YARN are you using. Since YARN 3.1.0, you can enable NUMA scheduling in config. Patch can be found here:
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Can multiple yarn containers inside same node share disk memory?
Does Spark on yarn deal with Data locality while launching executors
How YARN knows data locality in Apache spark in cluster mode
Yarn job memory untilization
What is Memory reserved on Yarn
run Spark-Submit on YARN but Imbalance (only 1 node is working)
Yarn job crashes with exit code 143 even though node has available memory
YARN Reserved Memory Issue
Spark Yarn Memory configuration
Yarn virtual memory usage
Related Tags