Python. Multiprocessing pool. How to obtain all results

Question

Started to play with Python and stuck with one silly situation. My function returns two values. Using MP I'd like to make execution faster.

import time
import multiprocessing as mp

def foo(bar, baz):
    print("start")
    time.sleep(2)
    print("end")
    return bar, baz


if __name__ == '__main__':
    pool = mp.Pool(processes=4)

    ret = []
    i = 0
    while i < 4 :
        j = i + 1
        while j < 4 :
            results = [pool.apply_async(foo, args=(i, j))]
            j = j + 1
        i = i + 1

    output = [p.get() for p in results]
    print(output)

In the same time finally I want to get a list of returned values like [(0,1),(0,2),...]. However I see only one pair instead. Any idea how to change this code shortly to have all results and do not lost parallelism? Thanks!

start
start
start
start
end
start
end
start
end
end
end
end
[(2, 3)]

Answer 1

You're overwriting your results list on each iteration, but you should be appending to it. This should do the trick:

results = []
while i < 4 :
    j = i + 1
    while j < 4 :
        results.append(pool.apply_async(foo, args=(i, j)))
        j = j + 1
    i = i + 1

output = [p.get() for p in results]

Answer 2

DIAGNOSIS

Here's your code with a little debugging, and using for statements for readability:

for i in range(4):
    for j in range(i+1, 4):
        results = [pool.apply_async(foo, args=(i, j))]
        print("TRACE", results]

output = [p.get() for p in results]
print("FINAL", output)

Looking at the output, you can see the problem: although you made 6 distinct result collectors, result is merely the most recent at any given time. Thus, you need to collect them all.

Output:

TRACE [<multiprocessing.pool.ApplyResult object at 0x7faf4b711390>]
TRACE [<multiprocessing.pool.ApplyResult object at 0x7faf4b7114e0>]
TRACE [<multiprocessing.pool.ApplyResult object at 0x7faf4b711588>]
TRACE [<multiprocessing.pool.ApplyResult object at 0x7faf4b711630>]
TRACE [<multiprocessing.pool.ApplyResult object at 0x7faf4b7116d8>]
start
TRACE [<multiprocessing.pool.ApplyResult object at 0x7faf4b711780>]
start
start
start
end
end
start
start
end
end
end
end
FINAL [(2, 3)]

---

SOLUTION

"Merely" use a nested-loop list comprehension to get the list of result applications you want:

results = [pool.apply_async(foo, args=(i, j))
              for i in range(4)
                 for j in range(i+1, 4)
          ]
print("TRACE", results)

output = [p.get() for p in results]
print("FINAL", output)

Output:

TRACE [<multiprocessing.pool.ApplyResult object at 0x7f52af945390>, 
       <multiprocessing.pool.ApplyResult object at 0x7f52af945438>, 
       <multiprocessing.pool.ApplyResult object at 0x7f52af9454e0>, 
       <multiprocessing.pool.ApplyResult object at 0x7f52af945588>,  
       <multiprocessing.pool.ApplyResult object at 0x7f52af945630>, 
       <multiprocessing.pool.ApplyResult object at 0x7f52af9456d8>]
start
start
start
start
end
end
end
end
start
start
end
end
FINAL [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3)]