weird behavior of multiprocessing scipy optimization inside of a function

Question

Here is a simple code that runs well. Even if the function minimize wraps the scipy.optimize.minimize it does not complain about pickling

import numpy as np
from scipy import optimize
from multiprocessing import Pool

def square(x):
    return np.sum(x**2+ 2*x)

def minimize(args):
    f,x = args
    res = optimize.minimize(f, x, method = 'L-BFGS-B')
    return res.x

x = np.random.rand(8,10)

args = [(square,x[i]) for i in range(8)]
p = Pool(8)
p.map(minimize,args)

However, if try the following it fails with the pickling error

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

def run():
    def square(x):
        return np.sum(x**2+ 2*x)

    def minimize(args):
        f,x = args
        res = optimize.minimize(f, x, method = 'L-BFGS-B')
        return res.x

    x = np.random.rand(8,10)

    args = [(square,x[i]) for i in range(8)]
    p = Pool(8)
    p.map(minimize,args)

run()

I want to make a module to use scipy minimize in parallel with many population of initial guesses. However, as shown in the example, when I make it a module, it fails.

Answer 1

The problem is that Python cannot pickle nested functions, and in your second example, you're trying to pass the nested minimize and square functions to your child process, which requires pickling.

If there's no reason that you must nest those two functions, moving them to the top-level of the module will fix the issue. You can also see this question for techniques to pickle nested functions.