I am writing a multiprocessing
routine to run on a server with plenty of CPUs. However, the server has multiple users and its usage may vary. So I would like to adapt the number of processors being used according to the current load.
multiprocessing.cpu_count()
multiprocessing.Pool(processes=no_cpus)
during activity, in case the load on the server has changed after a while? There are a number of complications...
Processes (and threads) are scheduled by the Linux kernel on any CPU. Even determining the "current CPU" is awkward -- see How can I see which CPU core a thread is running in?
multiprocessing.Pool
is designed to start up N workers, which run "forever." Each accepts a task from a queue, does some work, then outputs data. A Pool
doesn't change size. Two suggestions:
19:05:07 up 4 days, 20:43, 3 users, load average:
0.99 , 1.01, 0.82
The last three numbers are the "load average" over the last minute, five minutes, and 15 minutes. Consider using the first number to load-balance your application.
time.sleep(factor)
after completing each piece of work. Thus you can increase the factor when the system is busy (high load average), and make the delay shorter when the system is more idle (low load; ie surfing). Pool stays same size.
I followed johntellsall 's solution, here's some actual simple schematic. There is one, as Python (for me) confuses virtual with actual cpu. I've decided to calibrate against the average load of the past 15 minutes.
Sleeping numbers are quite arbitrary.
def sleepForMultiCore():
# divide by 2 since Python does not distinguish physical and virtual core
cores = 0.5*mp.cpu_count()
loadAvg = os.getloadavg()[2]
if loadAvg > cores*1.3:
sleepTime = 5*60
elif loadAvg > cores:
sleepTime = 2*60
elif loadAvg > cores*0.9:
sleepTime = 1*60
else:
sleepTime = 0
print ('sleeping for ', sleepTime)
time.sleep(sleepTime)
I think the best library for your question is psutil with which you can manage some CPU and Memory usage in some different OS':
psutil is a module providing an interface for retrieving information on running processes and system utilization (CPU, memory) in a portable way by using Python, implementing many functionalities offered by tools like ps, top and Windows task manager.
It currently supports Linux, OS X, FreeBSD and Windows with Python versions from 2.4 to 3.1 by using a unique code base.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.