在 Windows 上降低 multiprocessing.Pool 的进程优先级

Question

I use multiprocessing.Pool() to parallelize some heavy Pandas processing but find that it is a bit too successful. My CPU usage goes to 100% and my entire computer becomes very unresponsive. Even the mouse becomes difficult to use.

I can change the process priority of my process with this code.

import psutil
p = psutil.Process(os.getpid())
p.nice = psutil.BELOW_NORMAL_PRIORITY_CLASS

However, when I look in Windows Task Manager I find that only the main python.exe process has been changed to below normal priority.

Is there a good way to reduce the priority of the pool processes?

Answer 1

You can try setting priority of your process' children after you spawned them. Something like:

import psutil

# spawn children and/or launch process pool here

parent = psutil.Process()
parent.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)
for child in parent.children():
    child.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)

Answer 2

The same result as by using the answer by @Giampaolo Rodolà is achieved simply by setting the parent process priority before spawning the children:

import psutil

parent = psutil.Process()
parent.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)
# the rest of your code

The children processes will inherit the parent's priority. If, however, the parent is to be set to different priority than the children, then the code provided by @Giampaolo Rodolà is needed.

Answer 3

The Python documentation states that when a pool is created you can specify the number of processes. If you don't, it will default to os.cpu_count. Consequently, you get the expected behavior that all the available logical cores are used. In turn, the computer becomes unresponsive.

It would probably be better to do something simpler by just controlling the number of processes created. A rule of thumb is to reserve 2 to 4 logical cores for interactive processing.

Also, the Python documentation states " This number [os.cpu_count()] is not equivalent to the number of CPUs the current process can use. The number of usable CPUs can be obtained with len(os.sched_getaffinity(0)) "

There are several other details that need to be addressed. I have tried to capture them at this gist . All that you have to do is change LOGICAL_CORES_RESERVED_FOR_INTERACTIVE_PROCESSING for your particular use case.