使用梯度下降实现线性回归

Question

I have just started in machine learning and currently taking the course by andrew Ng's Machine learning Course. I have implemented the linear regression algorithm in python but the result is not desirable. I code of python is as follows:

import numpy as np

x = [[1,1,1,1,1,1,1,1,1,1],[10,20,30,40,50,60,70,80,90,100]]
y = [10,16,20,23,29,30,35,40,45,50]

x = np.array(x)
y = np.array(y)

theta = np.zeros((2,1))

def Cost(x,y,theta):
    m = len(y)
    pred_ions = np.transpose(theta).dot(x)
    J = 1/(2*m) * np.sum((pred_ions - y)*(pred_ions - y))
    return J

def GradientDescent(x,y,theta,iteration,alpha):
    m = len(y)
    pred_ions = np.transpose(theta).dot(x)
    i = 1
    while i <= iteration:
        theta[0] = theta[0] - alpha/m * np.sum(pred_ions - y)
        theta[1] = theta[1] - alpha/m * np.sum((pred_ions - y)*x[1,:])
        Cost_History = Cost(x,y,theta)
        i = i + 1

    return theta[0],theta[1]
itera = 1000
alpha = 0.01
a,b = GradientDescent(x,y,theta,itera, alpha)
print(a)
print(b)

I am not able to figure out what exactly is the problem. But, my results are something very strange. The value of parameter is, according to above code, are 298 and 19890. Any Help would be appreciated. Thanks.

Answer 1

Ah. I did this assignment too a while ago.

See this mentioned in Page 7 of the assignment PDF:

Octave/MATLAB array indices start from one, not zero. If you're storing θ0 and θ1 in a vector called theta, the values will be theta(1) and theta(2).

So, in your while loop, change the theta[0] and theta[1] to theta[1] and theta[2] . It should work right.

Also, if you are storing the Cost in Cost_History, shouldn't it include the iteration variable like

Cost_History[i] = Cost(x,y,theta)

Just check that too! Hope this helped.

Edit 1: Okay, I have understood the issue now. In his video, Andrew Ng says that you need to update both the thetas simultaneously. To do that, store the theta matrix in a temp variable. And update theta[0] and theta[1] based on the temp values.

Currently in your code, during theta[1] = it has changed the theta[0] to the newer value already, so both are not being updated simultaneously.

So instead, do this:

while i <= iteration:
    temp = theta
    theta[0] = theta[0] - alpha/m * np.sum(np.transpose(temp).dot(x) - y)
    theta[1] = theta[1] - alpha/m * np.sum((np.transpose(temp).dot(x) - y)*x[1,:])
    Cost_History[i] = Cost(x,y,theta)
    i = i + 1

It should work now, if not, let me know, I will debug on my side.