带有多个参数的python多重处理

Question

I'm trying to multiprocess a function that does multiple actions for a large file but I'm getting the knownle pickling error eventhough Im using partial .

The function looks something like this:

def process(r,intermediate_file,record_dict,record_id):

    res=0

    record_str = str(record_dict[record_id]).upper()
    start = record_str[0:100]
    end= record_str[len(record_seq)-100:len(record_seq)]

    print sample, record_id
    if r=="1":

        if something:
            res = something...
            intermediate_file.write("...")

        if something:
            res = something
            intermediate_file.write("...")



    if r == "2":
        if something:
            res = something...
            intermediate_file.write("...")

        if something:
            res = something
            intermediate_file.write("...")

    return res

The way im calling it is the following in another function:

def call_func():
    intermediate_file = open("inter.txt","w")
    record_dict = get_record_dict()                 ### get infos about each record as a dict based on the record_id
    results_dict = {}  
    pool = Pool(10)
    for a in ["a","b","c",...]:

        if not results_dict.has_key(a):
            results_dict[a] = {}

        for b in ["1","2","3",...]:

            if not results_dict[a].has_key(b):
                results_dict[a][b] = {}


            results_dict[a][b]['res'] = []

            infile = open(a+b+".txt","r")
            ...parse the file and return values in a list called "record_ids"...

            ### now call the function based on for each record_id in record_ids
            if b=="1":
                func = partial(process,"1",intermediate_file,record_dict)
                res=pool.map(func, record_ids)
                ## append the results for each pair (a,b) for EACH RECORD in the results_dict 
                results_dict[a][b]['res'].append(res)

            if b=="2":
                func = partial(process,"2",intermediate_file,record_dict)
                res = pool.map(func, record_ids)
                ## append the results for each pair (a,b) for EACH RECORD in the results_dict
                results_dict[a][b]['res'].append(res) 

    ... do something with results_dict...

The idea is that for each record inside the record_ids, I want to save the results for each pair (a,b).

I'm not sure what is giving me this error:

  File "/code/Python/Python-2.7.9/Lib/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/code/Python/Python-2.7.9/Lib/multiprocessing/pool.py", line 558, in get
    raise self._value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function faile

d

Answer 1

func is not defined at the top level of the code so it can't be pickled. You can use pathos.multiprocesssing which is not a standard module but it will work.

Or, use something diferent to Pool.map maybe a Queue of workers ? https://docs.python.org/2/library/queue.html

In the end there is an example you can use, it's for threading but is very similar to the multiprocessing where there is also Queues...

https://docs.python.org/2/library/multiprocessing.html#pipes-and-queues