How to run two parallel processes subsequently?

Question

I want to make sure the following two parallel processes are implemented one after the other. In particular, I want the ten f functions are implemented first, and after that part is finished, the ten g functions are implemented. Does anyone know how I should modify my code?

from multiprocessing import Process
import time
import random

wait_low = 0.1
wait_high = 15

def f(i):
    time.sleep(random.uniform(wait_low, wait_high))
    print 'hello'+str(i)

def g(i):
    time.sleep(random.uniform(wait_low, wait_high))
    print 'hey'+str(i)


if __name__ == '__main__':
    for j in range(10):
        p = Process(target=f, args=(j,))
        p.start()
    p.join()

    print "switch"

    # comment
    for j in range(10):
        q = Process(target=g, args=(j,))
        q.start()
    q.join()

    time.sleep(5)

    print "I'm done"

And the results I got are:

hello2
hello0
hello1
hello5
hello6
hello8
hello3
hello7
hello9
switch
hey6
hey3
hello4
hey9
hey8
I'm done
hey2
hey0
hey1
hey5
hey7
hey4

Thanks so much!

Answer 1

All f 's and g 's need to be joined.

if __name__ == '__main__':
    fs = []
    for j in range(10):
        p = Process(target=f, args=(j,))
        p.start()
        fs.append(p)

    for f in fs:
        f.join()

    print "switch"

    # comment
    gs = []
    for j in range(10):
        q = Process(target=g, args=(j,))
        q.start()
        gs.append(q)

    for g in gs:
        g.join()

    print "I'm done"

outputs:

hello2
hello8
hello5
hello6
hello9
hello1
hello4
hello3
hello7
hello0
switch
hey0
hey7
hey2
hey8
hey4
hey3
hey1
hey9
hey5
hey6
I'm done

Answer 2

Your problem is in your code you are only joining the last process you spawn in the loop, you could continue before previous ones are complete, that causes the interleaving of output.

You could use a process pool:

from multiprocessing.pool import Pool
import random
import time

wait_low = 0
wait_high=1
def f(i):
    time.sleep(random.uniform(wait_low, wait_high))
    return 'hello'+str(i)

def g(i):
    time.sleep(random.uniform(wait_low, wait_high))
    return 'hey'+str(i)


pool = Pool()
for output in pool.imap_unordered(f, range(10)):
    print output
for output in pool.imap_unordered(g, range(10)):
    print output

Answer 3

Use a blocking map function instead of doing the nitty-gritty work yourself. You can do the following with the built-in multiprocessing , but since I am lazy I just did it from the interpreter (doing so requires a fork of multiprocessing called pathos .

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> import time
>>> import random
>>> 
>>> wait_low = 0.1
>>> wait_high = 15
>>> 
>>> def f(i):
...   time.sleep(random.uniform(wait_low, wait_high))
...   print 'hello'+str(i)
... 
>>> def g(i):
...   time.sleep(random.uniform(wait_low, wait_high))
...   print 'hey'+str(i)
... 
>>> 

then create and launch the map

>>> p = Pool()
>>> r = p.map(f, range(10)); print "switch"; r = p.map(g, range(10)); print "I'm done"
hello6
hello2
hello1
hello0
hello5
hello8
hello3
hello4
hello7
hello9
switch
hey5
hey6
hey7
hey1
hey9
hey4
hey8
hey3
hey0
hey2
I'm done
>>> 

you can get pathos here: https://github.com/uqfoundation