不能多进程用户定义的代码-无法腌制

Question

I tried using multiprocessing on my task, which generally means to do some calculation and then pass back the result. The problem is that the code defining the calculation is defined by user, it is compiled from string before the execution. This works perfect using exec() , eval() or compile() etc. when being run in the main process. The example below works only for f1 function but not for f2 . I get 'Can't pickle class 'code'`. Is there any way round this? For example using multiprocessing differently? Or using other package? Or some more low level stuff? Unfortunatelly passing the string to the process and then compiling inside the process is not an option for me because of the design of the whole application (ie the code string is 'lost' and only the compiled version is available).

import multiprocessing

def callf(f, a):
    exec(f, {'a': a})

if __name__ == "__main__":
    f = compile("print(a)", filename="<string>", mode="exec")
    callf(f, 10)  # this works
    process = multiprocessing.Process(target=callf, args=(f, 20))  # this does not work
    process.start()
    process.join()

UPDATE: here is another attempt, which is actually closer to my actual need. It results in different error message, but also cannot pickle the function.

import multiprocessing

if __name__ == "__main__":
    source = "def f(): print('done')"
    locals = dict()
    exec(source, {}, locals)
    f = locals['f']
    f()  # this works
    process = multiprocessing.Process(target=f)  # this does not work
    process.start()
    process.join()

Answer 1

pickle can't serialize code objects but dill can. There is a dill-based fork of multiprocessing called multiprocessing_on_dill but I have no idea how good it is. You could also just dill-encode the code object to make standard multiprocessing happy.

import multiprocessing
import dill

def callf_dilled(f_dilled, a):
    return callf(dill.loads(f_dilled), a)

def callf(f, a):
    exec(f, {'a': a})

if __name__ == "__main__":
    f = compile("print(a)", filename="<string>", mode="exec")
    callf(f, 10)  # this works
    process = multiprocessing.Process(target=callf_dilled, 
        args=(dill.dumps(f), 20))  # now this works too!
    process.start()
    process.join()