Multiprocessing in Python 3 parallel processing and waiting on jobs

Question

I have a piece of code that queries a DB and returns a set of IDs. For each ID, I need to run a related query to get a dataset. I would like to run the queries in parallel to speed up the processing. Once all the processes are run, then I build a block of text and write that to a file, then move to the next id.

1. How do I ensure that all the processes start at the same time, then wait for all of them to complete before moving to the page =... and writefile operations?
2. If run as it, I get the following error: Process object is not iterable (on line 9).

Here is what I have so far:

from helpers import *
import multiprocessing

idSet = getIDset(10) 

for id in idSet:

ds1 = multiprocessing.Process(target = getDS1(id))
ds1list1, ds1Item1, ds1Item2 = (ds1)

    ds2 = multiprocessing.Process(target = getDS2(id))
    ds3 = multiprocessing.Process(target = getDS3(id))
    ds4 = multiprocessing.Process(target = getDS4(id))
    ds5 = multiprocessing.Process(target = getDS5(id))

    movefiles = multiprocessing.Process(moveFiles(srcPath = r'Z://', src = ds1Item2 , dstPath=r'E:/new_data_dump//'))

 ## is there a better way to get them to start in unison than this?
    ds1.start()
    ds2.start()
    ds3.start()
    ds4.start()
    ds5.start()

 ## how do I know all processes are finished before moving on?
    page = +ds1+'\n' \
           +ds2+'\n' \
           +ds3+'\n' \
           +ds4+'\n' \
           +ds5+'\n' 

    writeFile(r'E:/new_data_dump/',filename+'.txt',page)

Answer 1

I usually keep my "processes" in a list.

plist = []
for i in range(0, 5) :
    p = multiprocessing.Process(target = getDS2(id))
    plist.append(p)

for p in plist :
    p.start()


... do stuff ...


for p in plist :
    p.join() # <---- this will wait for each process to finish before continuing

Also I think you have an issue with creating your Process. "target" is supposed to be a function. Not the result of a function as it seems you have it (unless your function returns functions).

It should look like this:

p = Process(target=f, args=('bob',))

Where target is the function, and args is a tuple of arguemnts passed like so:

def f(name) :
    print name