fmin_cg在scipy中的梯度函数

Question

I am trying to use the conjugate gradient algorithm (fmin_cg) from scipy to find parameters theta which give the best fit within a linear model.

Data file HouseData.csv (eg house area, house price):

120, 250
200, 467
250, 500
1200, 2598
1500, 3000

The code is:

from scipy import optimize
import numpy as np

data=np.genfromtxt('HouseData.csv',delimiter=',')
X=np.c_[np.ones(len(data)),data[:,:-1]]
Y=data[:,[-1]]

def cost_Function(theta):
    theta1=theta[np.newaxis].T
    #print('theta: ',theta1)
    cost = Y-np.dot(X, theta1)
    return (cost*cost).sum()

# Gradient Function
def gradf(theta):
    theta1 = theta[np.newaxis].T
    cost = Y - np.dot(X, theta1)
    #print('cost*X.sum(0) is', np.sum(cost*X,axis=0))
    return np.sum(cost*X,axis=0)


x0 = np.asarray((0,1)) #initial guess
result = optimize.fmin_cg(cost_Function,x0,fprime=gradf)
print(result)    

Without fprime=gradf the code returns the correct result, but what is the problem with the gradient function? When including it as above, the algorithm returns exactly the input for theta. Is there anything else you would implement differently to improve performance? This is just a simple example but the algorithms should also run with X having many columns and rows.

(python 3.5.1, scipy and numpy most recent version)

Answer 1

Your gradient is clearly wrong.

Since your cost function is quadratic, we can approximate the gradient reasonably well with: gradf(x) = (f(x + eps) - f(x - eps)) / (2 eps) . Let's try that:

e0 = np.array([1, 0])
e1 = np.array([0, 1])
eps = 1e-5

x0 = np.array([1, 1])

df_yours = gradf(x0)
# array([  3.54000000e+03,   4.05583000e+06])

df_approx = np.array([
    cost_Function(x0 + eps*e0) - cost_Function(x0 - eps*e0),
    cost_Function(x0 + eps*e1) - cost_Function(x0 - eps*e1)
]) / (2 * eps)
# array([ -7.07999999e+03,  -8.11166000e+06])

Without doing mathematical analysis (which by the way, you absolutely should be doing rather than guessing ), your gradient function is off by a factor of -0.5 . That negative is pretty critical.

Answer 2

Eric's comment regarding the sign of the gradient function was crucial. Here is the currectly working code, where np.dot(X, theta1) - Y is now correct and a factor of 0.5 was added to cost_Function

from scipy import optimize
import numpy as np

data=np.genfromtxt('HouseData.csv',delimiter=',')
X=np.c_[np.ones(len(data)),data[:,:-1]]
Y=data[:,[-1]]

def cost_Function(theta):
    theta1=theta[np.newaxis].T
    cost = Y-np.dot(X, theta1)
    return 0.5*(cost*cost).sum()

# Gradient Function
def gradf(theta):
    theta1 = theta[np.newaxis].T
    cost = np.dot(X, theta1) - Y
    return np.sum(cost*X,axis=0)

x0 = np.asarray((0.1,2)) #initial guess

result = optimize.fmin_cg(cost_Function,x0,fprime=gradf)
print(result)