scipy.optimize.fmin的朴素并行化

Question

The function scipy.optimize.fmin_bfgs allows the user to input both a target function and a gradient. Since I have an 8-core machine on my desktop, I thought I could parallelize the solver by running

from scipy import optimize
import itertools
import numpy as np

def single_grad_point((idx,px)):
    p = px.copy()
    epsilon = 10**(-6.0)
    p[idx] += epsilon
    d1 = err_func(p)
    p[idx] -= 2*epsilon
    d2 = err_func(p)
    return (d1-d2)/(2*epsilon)

def err_func_gradient(p):
    P = multiprocessing.Pool()   
    input_args = zip(*(xrange(len(p)), itertools.cycle((p,))))
    sol = P.imap(single_grad_point, input_args)
    return np.array(list(sol))

optimize.fmin_bfgs(err_func, p0, fprime=err_func_gradient)

In a nutshell, I'm using multiprocessing to compute each direction of the gradient. If the target function err_func is expensive, this seems to gain substantial speedup. My question however is about the usage and con/destruction of all the multiprocessing.Pools . Since it's possible that err_func_gradient can be called tens of thousands of times, will this cause a slowdown or leak somewhere ?

Answer 1

You could use mystic , which provides parallel versions of a few of the scipy.optimize algorithms, including fmin and friends.

Trying to do a naive call to have each of the simplex evaluated in parallel is usually going to slow you down, unless you have some very very expensive objective functions to calculate. However, if you instead call several instances of fmin , you can actually get pseduo-GLOBAL optimization at steepest-descent speeds. The following example demonstrates an algorithm that has been used in several pubs (see below): https://github.com/uqfoundation/mystic/blob/master/examples/buckshot_example06.py

Or similarly, look at the examples here: using a fork of multiprocessing : https://github.com/uqfoundation/pathos/blob/master/examples2/optimize_cheby_powell_mpmap.py

or a fork of parallelpython (distributed parallel computing): https://github.com/uqfoundation/pathos/blob/master/examples2/optimize_cheby_powell_ppmap.py

or using an extension of mpi4py : https://github.com/uqfoundation/pathos/blob/master/examples2/optimize_cheby_powell_mpimap.py

Get mystic (the solver framework) and pathos (the parallel computing framework) here: https://github.com/uqfoundation

Pub references (both slightly out of date): http://conference.scipy.org/proceedings/scipy2011/mckerns.html http://trac.mystic.cacr.caltech.edu/project/mystic/wiki/Publications

If, however, you wanted to do the more naive version of fmin , the best approach is to only initialize and load the pool once. This is what pathos provides for you already, but if you wanted to code it yourself, just save the instance of the pool as a singleton. https://github.com/uqfoundation/pathos/blob/master/pathos/multiprocessing.py

Answer 2

For starters, you can calc the gradient as

(f(x)-f(x+eps))/eps

Then calc f(x) once for all partial derivatives. That should save you half the work