为什么此代码中的python多处理速度较慢？

Question

I have this sample code. I'm making some math computation (graph theory) and I want to increase the speed so I decided to go with multiprocessing but surprisingly the same code runs even slower than the single process version.

I expect that if you have a list and you split it in half and start two processes it should take approximately half time. I don't think I have some synchronization issue so why the hell is this so slow?

from multiprocessing import Process, Queue

def do_work(queue, aList):

    for elem in aList:
        newList.append(a_math_function(elem))

    queue.put(max(newList))
    return max(newList)

def foo():

    #I have let's say aList with 100000 objects

    q1 = Queue(); q2 = Queue()

    p1 = Process(target=do_work, args=[q1, aList[0: size/2])])
    p2 = Process(target=do_work, args=[q2, aList[size/2+1: size])])

    p1.start(); p2.start()
    p1.join(); p2.join()

    print(max(q1.get(),q2.get())

Answer 1

Instead of using multiprocessing, try cutting down on the amount of allocation you're doing. You're just filling RAM up with temporary lists (for example the halves of aList you pass the processes, and then the newList lists you create in each process). Quite possibly this is the cause of the slowness as lots of allocation means lots of garbage collection.

Try replacing all your code with this and measuring the speed.

def foo():
    print max(a_math_function(x) for x in aList)

Answer 2

Use a multiprocessing.Pool.map() to distribute the workload between workers and get the result aggregated from it.

See the example here below http://docs.python.org/library/multiprocessing.html#multiprocessing.pool.AsyncResult.successful