Lagrange multipler BFGS optimization with python

Question

I would like to use the scipy optimization routines, in order to minimize functions while applying some constraints. I would like to apply the Lagrange multiplier method, but I think that I missed something.

My simple example: minimize f(x,y)=x^2+y^2, while keeping the constraint: y=x+4.0

import numpy as np
from scipy.optimize import fmin_bfgs
#X=[x,y,l]

def f(X):
   x=X[0]
   y=X[1]
   return x**2+y**2

def g(X):
   x=X[0]
   y=X[1]
   return y-x-4.000

def L(X):
   l=X[2]
   return f(X)+l*g(X)


def dL(X):
   x=X[0]
   y=X[1]
   l=X[2]
   gx=2.0*x
   gy=2.0*y
   gl=g(X)
   tmp=np.array([gx,gy,gl])
   return tmp


x0=np.array([-2.0,2.0,0.0])



print "f(x0)\t\t g(x0) \t\t L(x0)"
print "%12.8f\t%12.8f\t%12.8f\t"%(f(x0),g(x0),L(x0))
print "dL(x0)"
print dL(x0)


xopt=fmin_bfgs(L,x0,fprime=dL,disp=True)
print xopt

Even if my x0 is on the spot, the optimization diverges, badly. Could someone please explain me how one should include the Lagrange multiplier properly and how one should initialize the multiplier?

Answer 1

The main idea behind the Lagrangian multiplier is to construct a new objective function with the constrains already embedded. First you solve the equations to find the value of the multiplier and then optimize the new objetive function( http://www.math.vt.edu/people/mcquain/1526_Lag_opt_2012.pdf ). However in your example you're trying to find the value of the multiplier by optimization. By following the instructions provided in the lik you can find that the lagrangian multiplier for your problem is 4.

def f(X):
    x=X[0]
    y=X[1]
    return x**2+y**2

def g(X):
    x=X[0]
    y=X[1]
    return y-x-4.000

def L(X):
    return f(X)-4*g(X)

x0=np.array([0,0])
xopt=fmin_bfgs(L,x0,disp=True)

Hope it helps