[英]why my python multiprocessing can't use all cpu?
i have a server with 48 cores, so i decide to use python multiprocess to run my code with 46 process.我有一个有 48 个内核的服务器,所以我决定使用 python 多进程来运行我的 46 个进程的代码。
here is the multiprocessing wrapper function:这是多处理包装器 function:
import multiprocessing
def MPRun(func, args, dicts={}, n_pool=max(1, multiprocessing.cpu_count()-2), ret=True, show_process=True, show_num=100):
print('submit a multiprocess task with cpu', n_pool)
q = Queue()
if not isinstance(args, list) or not isinstance(dicts, dict):
print('args should be list, dicts should be dict', type(args), type(dicts))
sys.exit(1)
pool = multiprocessing.Pool(processes=n_pool)
res = []
for i, arg in enumerate(args):
if not isinstance(args[0], list):
q.put(pool.apply_async(func, tuple([arg]), dicts))
else:
q.put(pool.apply_async(func, tuple(arg), dicts))
count = 0
r = []
while count < len(args):
r.append(q.get().get())
count += 1
if show_process and count % show_num == show_num - 1:
print('MPRUN process[%d/%d]'%(count, len(args)))
pool.close()
pool.join()
return r
I call it like this:我这样称呼它:
MPRun(myfunc, [[task] for task in tasks])
when i call top command in linux, i found it do has 46 python process, but among them, only 8-12 is running, the left's state are s(which means wait cpu to run).当我在 linux 中调用 top 命令时,我发现它确实有 46 个 python 进程,但其中只有 8-12 个正在运行,左边的 state 是 s(等待 cpu 运行)。 and many cpu's idle is 100%(which means it is idle)许多cpu的空闲是100%(这意味着它是空闲的)
there is no other code running, so, i am confused, why those idle cpu dont run the state s tasks?没有其他代码在运行,所以,我很困惑,为什么那些空闲的 cpu 不运行 state 的任务?
Is there any problems with my MPRun()?我的 MPRun() 有什么问题吗? or is it related with my task type?还是与我的任务类型有关?
You didn't show what your tasks actually do.您没有显示您的任务实际做什么。 There could be many reasons why they are not using full CPU.他们没有使用完整 CPU 的原因可能有很多。 's' - means that the task is waiting on something to happen it could be one of the following: 's' - 表示任务正在等待某事发生,它可能是以下之一:
Try to profile your processes with 'strace' and see if there something going on there.尝试使用“strace”分析您的流程,看看那里是否发生了什么。
Also look at your /proc/interrupts to see if you have unsually high IO activity.还要查看您的 /proc/interrupts 以查看您是否有异常高的 IO 活动。 You have to take a baseline first to determine the interrupts volume on your system when it's idle, and then sample it again while your program is running您必须首先获取基线以确定系统空闲时的中断量,然后在程序运行时再次对其进行采样
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.