python multiprocessing函数返回的多个输出

Question

I am trying to use multiprocessing to return a list, but instead of waiting until all processes are done, I get several returns from one return statement in mp_factorizer, like this:

None
None
(returns list)

in this example I used 2 threads. If I used 5 threads, there would be 5 None returns before the list is being put out. Here is the code:

def mp_factorizer(nums, nprocs, objecttouse):
    if __name__ == '__main__':
        out_q = multiprocessing.Queue()
        chunksize = int(math.ceil(len(nums) / float(nprocs)))
        procs = []
        for i in range(nprocs):
            p = multiprocessing.Process(
                    target=worker,                   
                    args=(nums[chunksize * i:chunksize * (i + 1)],
                          out_q,
                    objecttouse))
            procs.append(p)
            p.start()

        # Collect all results into a single result dict. We know how many dicts
        # with results to expect.
        resultlist = []
        for i in range(nprocs):
            temp=out_q.get()
            index =0
            for i in temp:
                resultlist.append(temp[index][0][0:])
                index +=1

        # Wait for all worker processes to finish
        for p in procs:
            p.join()
            resultlist2 = [x for x in resultlist if x != []]
        return resultlist2

def worker(nums, out_q, objecttouse):
    """ The worker function, invoked in a process. 'nums' is a
        list of numbers to factor. The results are placed in
        a dictionary that's pushed to a queue.
    """
    outlist = []
    for n in nums:        
        outputlist=objecttouse.getevents(n)
        if outputlist:
            outlist.append(outputlist)   
    out_q.put(outlist)

mp_factorizer gets a list of items, # of threads, and an object that the worker should use, it then splits up the list of items so all threads get an equal amount of the list, and starts the workers. The workers then use the object to calculate something from the given list, add the result to the queue. Mp_factorizer is supposed to collect all results from the queue, merge them to one large list and return that list. However - I get multiple returns.

What am I doing wrong? Or is this expected behavior due to the strange way windows handles multiprocessing? (Python 2.7.3, Windows7 64bit)

EDIT: The problem was the wrong placement of if __name__ == '__main__': . I found out while working on another problem, see using multiprocessing in a sub process for a complete explanation.

Answer 1

if __name__ == '__main__' is in the wrong place. A quick fix would be to protect only the call to mp_factorizer like Janne Karila suggested:

if __name__ == '__main__':
    print mp_factorizer(list, 2, someobject)

However, on windows the main file will be executed once on execution + once for every worker thread, in this case 2. So this would be a total of 3 executions of the main thread, excluding the protected part of the code.

This can cause problems as soon as there are other computations being made in the same main thread, and at the very least unnecessarily slow down performance. Even though only the worker function should be executed several times, in windows everything will be executed thats not protected by if __name__ == '__main__' .

So the solution would be to protect the whole main process by executing all code only after if __name__ == '__main__' .

If the worker function is in the same file , however, it needs to be excluded from this if statement because otherwise it can not be called several times for multiprocessing.

Pseudocode main thread:

# Import stuff
if __name__ == '__main__':
    #execute whatever you want, it will only be executed 
    #as often as you intend it to
    #execute the function that starts multiprocessing, 
    #in this case mp_factorizer()
    #there is no worker function code here, it's in another file.

Even though the whole main process is protected, the worker function can still be started, as long as it is in another file.

Pseudocode main thread, with worker function:

# Import stuff
#If the worker code is in the main thread, exclude it from the if statement:
def worker():
    #worker code
if __name__ == '__main__':
    #execute whatever you want, it will only be executed 
    #as often as you intend it to
    #execute the function that starts multiprocessing, 
    #in this case mp_factorizer()
#All code outside of the if statement will be executed multiple times
#depending on the # of assigned worker threads.

For a longer explanation with runnable code, see using multiprocessing in a sub process

Answer 2

Your if __name__ == '__main__' statement is in the wrong place. Put it around the print statement to prevent the subprocesses from executing that line:

if __name__ == '__main__':
    print mp_factorizer(list, 2, someobject)

Now you have the if inside mp_factorizer , which makes the function return None when called inside a subprocess.