Python多处理包装器

Question

I have a simple multiprocessing wrapper. Which takes a function and runs it in parallel and get the data back.

However, I seem to be hard-coding my arguments (see apply() ) . I am not sure how I can pass a more dynamic argument list.

Here is my code

import multiprocessing as mp                                                     
import time                                                                      

class MPWrap(object):                                                            
    def __init__(self,verbose=False):                                            
        self.a=mp.Pool(10)                                                       
        self.resultObj=[]                                                        
        self.verbose=verbose                                                     

    def apply(self,func,args):                                                   
        for i in args:                                                           
            self.resultObj.append(                                               
                    self.a.apply_async(func,kwds={'x':i})                        
                )                                                                

    def status(self):                                                            
        incomplete_count = sum(1 for x in self.resultObj if not x.ready())       
        if self.verbose:                                                         
          s="todo %d/%d tasks"%(incomplete_count,len(self.resultObj))            
          print s                                                                
        return incomplete_count                                                  

    def get(self):                                                               
        return  [ i.get() for i in self.resultObj ]                              

    def __del__(self):                                                           
        self.a.close()                                                           
        self.a.join()                                                            
        print "Done..."                                                          



def square(x):                                                                   
    time.sleep(x)                                                                
    return x*x                                                                   

def main():                                                                      
    x=MPWrap()                                                                   

    x.apply(square,                                                              
            args=[1,2,3,4,5,6,7,8,9]                                             
            )                                                                    
    while x.status() is None:                                                    
        time.sleep(.5)                                                           
        if x.status()==0:break                                                   
    print x.get()  

if __name__=="__main__":                                                                      x[1, 4, 9, 16, 25, 36, 49, 64, 81]
    main()         

Answer 1

Using functions to generate function arguments should work:

def apply(self,func,args,fargs=lambda i:(),fkwargs=lambda i:{}):                                                   
    for i in args:                                                           
        self.resultObj.append(                                               
                self.a.apply_async(func,fargs(i),fkwargs(i))                        
            )

and

x.apply(square,                                                              
        args=[1,2,3,4,5,6,7,8,9],
        fkwargs=lambda i: {'x':i}                                             
        ) 

Answer 2

apply_async is to run independent single tasks. If you have a group of identical tasks you should be using map_async or one of its variants. I'm not actually sure what your wrapper is providing that doesn't already come from map_async .

Consider:

def main():
    x = Pool(10)
    res = x.map_async(square, range(1,10))
    while not res.ready():
        res.wait(0.5)
    print res.get()

ETA: If you want to generalise the arguments through the map function I can see some value in creating a wrapper for that purpose. Consider maybe:

def unwrap(fdesc):
    (f, args, kwargs) = fdesc
    return (f, f(*args, **kwargs))

You can then call x.map_async(unwrap, [(square, [i], {}) for i in range(1,10)]) and the results returned are in a tuple keyed with the function (so you can differentiate and interpret the results if you are using multiple functions).

Essentially with this you would create a list of tuples containing the function, the positional arguments, and the key-word arguments, and then you feed in that whole list to the one call to map_async .