[英]Do I benefit from the 96 vCPUs of a gcloud Compute Engine if in python multiprocessing.cpu_count() returns 64?
I am running a python parallel CPU intensive task on Google Compute Engine. 我在Google Compute Engine上运行python并行CPU密集型任务。 Hence, the greater the number of vCPUs I can run it on, the greater the speed. 因此,我可以运行它的vCPU数量越多,速度就越快。
I've read that there is no point in creating a multiprocessing pool with greater size than the number of available vCPUs, which makes sense, so I determine the size of my multiprocessing.dummy.Pool
pool using multiprocessing.cpu_count()
. 我已经读过创建一个大小超过可用vCPU数量的多处理池没有意义,这是有意义的,所以我使用multiprocessing.cpu_count()
确定multiprocessing.dummy.Pool
池的大小。
I am running this script in a Pod, using gcloud Kubernetes Engine, and tested on machines with less than 96 vCPUs during development. 我使用gcloud Kubernetes Engine在Pod中运行此脚本,并在开发期间在少于96个vCPU的计算机上进行测试。 The pool size automatically determined seemed always to match to the number of vCPUs. 自动确定的池大小似乎总是与vCPU的数量相匹配。 However, running it on a machine with 96 vCPUs, multiprocessing.cpu_count()
returns 64 and not 96. I don't care setting that size manually to 96 but the question is, will I benefit from those extra 32 vCPUs if python is not "aware" of them? 但是,在具有96个vCPU的计算机上运行它, multiprocessing.cpu_count()
返回64而不是96.我不在乎手动将该大小设置为96但问题是,如果python不是,我将从这些额外的32个vCPU中受益“意识到”他们?
The machine is a n1-highcpu-96 (96 vCPUs, 86.4 GB memory) running the Container-Optimized OS (cos). 该机器是运行容器优化OS(cos)的n1-highcpu-96(96个vCPU,86.4 GB内存)。 Python version is 3.6.3. Python版本是3.6.3。
There is an answer in the message board that someone linked to in a comment to the question, however, it seems better to have the answer on this page, as well as some explanation. 在留言板上有一个答案,有人在问题的评论中链接,但是,在这个页面上得到答案似乎更好,以及一些解释。
The short answer: inside a pod, run grep -c ^processor /proc/cpuinfo
- this number should agree with multiprocessing.cpu_count()
. 简短的回答:在pod中运行grep -c ^processor /proc/cpuinfo
- 这个数字应该与multiprocessing.cpu_count()
。 If it does, you can trust multiprocessing.cpu_count()
. 如果是,则可以信任multiprocessing.cpu_count()
。
However, AFAICT, this identifies all the cores on the node and completely ignores the resource limits set in your Kubernetes deployment YAML. 但是,AFAICT会识别节点上的所有核心,并完全忽略Kubernetes部署YAML中设置的资源限制。 For example, your deployment file might contain: 例如,您的部署文件可能包含:
spec:
containers:
- image: IMAGENAME
name: LABEL
ports:
- containerPort: 5000
resources:
limits:
cpu: 100m
memory: 400M
requests:
cpu: 50m
memory: 200M
In this article , the author gives the following function, which respects the resource limits (not requests): 在本文中 ,作者给出了以下功能,它尊重资源限制 (而不是请求):
import math
from pathlib import Path
def get_cpu_quota_within_docker():
cpu_cores = None
cfs_period = Path("/sys/fs/cgroup/cpu/cpu.cfs_period_us")
cfs_quota = Path("/sys/fs/cgroup/cpu/cpu.cfs_quota_us")
if cfs_period.exists() and cfs_quota.exists():
# we are in a linux container with cpu quotas!
with cfs_period.open('rb') as p, cfs_quota.open('rb') as q:
p, q = int(p.read()), int(q.read())
# get the cores allocated by dividing the quota
# in microseconds by the period in microseconds
cpu_cores = math.ceil(q / p) if q > 0 and p > 0 else None
return cpu_cores
So, for the example YAML, the division yields 0.1
, but b/c of the call to ceil
, it returns 1.0
. 因此,对于示例YAML,除法产生0.1
,但是对ceil
的调用的b / c,它返回1.0
。 So what you may be looking for is something like the following (assuming that you have the above-defined function get_cpu_quota_within_docker
defined): 所以你可能正在寻找的是类似下面的东西(假设你定义了上面定义的函数get_cpu_quota_within_docker
):
import multiprocessing
from somewhere import get_cpu_quota_within_docker
docker_cpus = get_cpu_quota_within_docker()
cpu_count = docker_cpus if docker_cpus else multiprocessing.cpu_count()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.