Python多处理行为，进程数大于内核数

Question

I am trying to use Python's multiprocessing library and would like to understand its behavior with different numbers of processes.

I hypothesized that there is no benefit from setting the processes larger than the number of cores. Contrary to the hypothesis, the experiment code below says that the computation time decreases even with processes above the number of cores (in my case, 4).

Can someone explain what is going on behind the scene and give some practical guide on how to set the number of processes?

from multiprocessing import Pool, cpu_count
import time
from datetime import datetime

My CPU counts.

cpu_count()
# 4

An experimental task, which takes approximately 0.5 second.

def f(x):
    time.sleep(0.5)
    return x*x

def execute_time(processes):
    t1 = datetime.now()
    with Pool(processes) as p:
        p.map(f, list(range(36)))
    t2 = datetime.now()
    return t2 - t1

for p in range(1, 25):
    t = execute_time(p)
    print(p, ":", t)

Yielded:

# 1 : 0:00:18.065411
# 2 : 0:00:10.051516
# 3 : 0:00:06.057016
# 4 : 0:00:04.562439
# 5 : 0:00:04.069810
# 6 : 0:00:03.173502
# 7 : 0:00:03.065977
# 8 : 0:00:03.082625
# 9 : 0:00:02.092880
# 10 : 0:00:02.090963
# 11 : 0:00:02.061613
# 12 : 0:00:01.704716
# 13 : 0:00:01.704880
# 14 : 0:00:01.615440
# 15 : 0:00:01.625117
# 16 : 0:00:01.621259
# 17 : 0:00:01.639741
# 18 : 0:00:01.236108
# 19 : 0:00:01.250113
# 20 : 0:00:01.255697
# 21 : 0:00:01.253459
# 22 : 0:00:01.260632
# 23 : 0:00:01.262124
# 24 : 0:00:01.247772

It makes sense that the function takes 18 seconds with one process (36 * 0.5 sec = 18 sec). So does the case with four processes (18 sec / 4 = 4.5 sec). But I am surprised that the computation time decreases with larger number of processes.

Answer 1

As Michael Butscher says in the comment, it was a special behavior for sleep . With a CPU intensive task, I saw the benefit is bounded at processes equaling the number of cores.

def f(x):
    out = 0
    for i in range(5000000):
        out += i
    return x*x

def execute_time(processes):
    t1 = datetime.now()
    with Pool(processes) as p:
        p.map(f, list(range(36)))
    t2 = datetime.now()
    return t2 - t1

for p in range(1, 25):
    t = execute_time(p)
    print(p, ":", t)

Gets:

# 1 : 0:00:13.329320
# 2 : 0:00:07.528552
# 3 : 0:00:09.943043
# 4 : 0:00:07.756005
# 5 : 0:00:08.262304
# 6 : 0:00:07.653659
# 7 : 0:00:07.677038
# 8 : 0:00:07.591766
# 9 : 0:00:07.502283
# 10 : 0:00:07.710826
# 11 : 0:00:06.006874
# 12 : 0:00:09.720279
# 13 : 0:00:07.912836
# 14 : 0:00:07.616807
# 15 : 0:00:07.740225
# 16 : 0:00:07.721783
# 17 : 0:00:07.836259
# 18 : 0:00:07.665993
# 19 : 0:00:07.564645
# 20 : 0:00:07.653607
# 21 : 0:00:07.754377
# 22 : 0:00:07.886036
# 23 : 0:00:11.696323
# 24 : 0:00:07.674243