[英]Python multiprocessing context switching of CPU's
I have created this simple code to check multiprocessing reading from a global dictionary object:我创建了这个简单的代码来检查全局字典 object 中的多处理读取:
import numpy as np
import multiprocessing as mp
import psutil
from itertools import repeat
def computations_x( max_int ):
#random selection
mask_1 = np.random.randint( low=0, high=max_int, size=1000 )
mask_2 = np.random.randint( low=0, high=max_int, size=1000 )
exponent_1 = np.sqrt( np.pi )
vector_1 = np.array( [ read_obj[ k ]**( exponent_1 ) for k in mask_1 ] )
vector_2 = np.array( [ read_obj[ k ]**np.pi for k in mask_2 ] )
result = []
for j in range(100):
res_col = []
for i in range(100):
c = np.multiply( vector_1, vector_2 ).sum( axis=0 )
res_col.append(c)
res_col = np.array( res_col )
result.append( res_col )
result = np.array( result )
return result
global read_obj
total_items = 40000
max_int = 1000
keys = np.arange(0, max_int)
number_processors = psutil.cpu_count( logical=False )
#number_used_processors = 1
number_used_processors = number_processors - 1
number_tasks = number_used_processors
read_obj = { k: np.random.rand( 1000 ) for k in keys }
pool = mp.Pool( processes = number_used_processors )
args = list( repeat( max_int, number_tasks ) )
results = pool.map( computations_x, args )
pool.close()
pool.join()
However, when looking at CPU performance, I see that the CPU's are being switched by the OS when performing the computations.但是,在查看 CPU 性能时,我发现在执行计算时操作系统正在切换 CPU。 I am running on Ubuntu 18.04, is this normal behaviour when using Python's MP module?我在 Ubuntu 18.04 上运行,这是使用 Python 的 MP 模块时的正常行为吗? Here is what I observe in the system monitor when debugging the code (I am using Eclipse2019 for debugging)这是我在调试代码时在系统监视器中观察到的(我使用Eclipse2019进行调试)
Any help is appreciated, as in my main project I need to share a global "read only" object through processes in the same spirit as is done here, and I want to be sure this is not affecting performance really badly;感谢您提供任何帮助,因为在我的主要项目中,我需要以与此处相同的精神通过流程共享全局“只读” object,并且我想确保这不会严重影响性能; I also want to make sure all tasks are executed concurrently within the Pool class.我还想确保在池 class 中同时执行所有任务。 thanks.谢谢。
I'd say that is the normal behaviour as the OS has to make sure that other processes are not starving for CPU time.我会说这是正常行为,因为操作系统必须确保其他进程不会因 CPU 时间而饿死。
Here's a nice article on the OS scheduler basics: https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part1.html这是一篇关于操作系统调度程序基础的好文章: https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part1.html
It's focusing on Golang but the first part is pretty general.它专注于 Golang,但第一部分非常笼统。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.