python, better syntax with multiprocesses

Question

I have the following method:

def GetMarketData(fr, cr, dct1, dct2, dict3, dct5, dct5):
    md = MarketData()

    q1 = Queue()
    q2 = Queue()
    q3 = Queue()
    q4 = Queue()
    q5 = Queue()
    q6 = Queue()
    q7 = Queue()
    p1 = Process(target=md.GetMD1, args=(q1,))
    p2 = Process(target=md.GetMD2, args=(q2,))
    p3 = Process(target=md.GetMD3, args=(q3,))
    p4 = Process(target=md.GetMD4, args=(q4,))
    p5 = Process(target=md.GetMD5, args=(q5,))
    p6 = Process(target=md.GetMD6, args=(q6,))
    p7 = Process(target=md.GetMD7, args=(q7,))
    p1.start()
    p2.start()
    p3.start()
    p4.start()
    p5.start()
    p6.start()
    p7.start()
    fr.append(q1.get())
    dct1.update(q2.get())
    dct2.update(q3.get())
    dct3.update(q4.get())
    cr.append(q5.get())
    dct4.update(q6.get())
    dct5.update(q7.get())
    p1.join()
    p2.join()
    p3.join()
    p4.join()
    p5.join()
    p6.join()
    p7.join()

    #print "good"

Is there a good way to rewrite this in a few line as possible. If I would need to open the same process over each queue that would not be a problem, but I need to run several different programs in parallel and do not see how to rewrite this in a fewer lines...

Thanks !!!

Answer 1

As a first simplification, I'd use lists:

def GetMarketData(fr, cr, dct1, dct2, dict3, dct5, dct5):
    md = MarketData()

    ques = [Queue() for _ in range(7)]
    procs = [Process(target=getattr(md,'GetMD%d'%(i),args=(q,)) 
             for i,q in enumerate(ques,1)]
    for p in procs:
        p.start()

    fr.append(ques[0].get())
    dct1.update(ques[1].get())
    dct2.update(ques[2].get())
    dct3.update(ques[3].get())
    cr.append(ques[4].get())
    dct4.update(ques[5].get())
    dct5.update(ques[6].get())

    for p in procs:
        p.join()

Answer 2

Instead of:

q1 = Queue()
q2 = Queue()
q3 = Queue()
q4 = Queue()
q5 = Queue()
q6 = Queue()
q7 = Queue()

You can instead write:

queues = [Queue() for i in xrange(7)]

Now your queues are queues[0] through queues[6] . You can do the same thing with your processes:

processes = [Process(target=getattr("md.GetMD%d" % (i+1)), args=(queues[i],))
             for i in xrange(7)]

And then start them using a loop:

for process in processes:
    process.start()

Similarly for joining later:

for process in processes:
    process.join()

That leaves this section:

fr.append(q1.get())
dct1.update(q2.get())
dct2.update(q3.get())
dct3.update(q4.get())
cr.append(q5.get())
dct4.update(q6.get())
dct5.update(q7.get())

Which could be loopified in a similar manner just by accepting any number of arguments in your function rather than named arguments; then we can loop over the arguments:

for queue, result_dict in zip(queues, result_dicts):
    result_dict.update(queue.get())

Then we can also just make it so we pass in any number of dictionaries, so rather than using the magic number of 7 we just create as many queues and processes as arguments. The function doesn't need to know that some dictionaries are different from others because in fact they are all processed identically anyway.

Putting it together, we get:

def GetMarketData(*result_dicts):

    queues = [Queue() for item in result_dicts]

    processes = [Process(target=getattr("md.GetMD%d" % (i+1)), args=(queues[i],))
                 for i in xrange(len(result_dicts))]

    for process in processes:
        process.start()

    for queue, result_dict in zip(queues, result_dicts):
        result_dict.update(queue.get())

    for process in processes:
        process.join()

(Unrelated note: I've not done much with multiprocessing , but wouldn't you want to join the processes before reading from the queues, to make sure the results in the queues are complete?)