使用梯度下降算法进行线性回归，获得意外结果

Question

I'm trying to create a function which returns the value of θ 0 & θ 1 of hypothesis function of linear regression. But I'm getting different results for different initial (random) values of θ 0 & θ 1 .

What's wrong in the code?

training_data_set = [[1, 1], [2, 3], [4, 3], [3, 2], [5, 5]]
initial_theta = [1, 0]


def gradient_descent(data, theta0, theta1):
    def h(x, theta0, theta1):
        return theta0 + theta1 * x

    m = len(data)
    alpha = 0.01

    for n in range(m):
        cost = 0
        for i in range(m):
            cost += (h(data[i][0], theta0, theta1) - data[i][1])**2

        cost = cost/(2*m)

        error = 0
        for i in range(m):
            error += h(data[i][0], theta0, theta1) - data[i][1]

        theta0 -= alpha*error/m
        theta1 -= alpha*error*data[n][0]/m

    return theta0, theta1


for i in range(5):
    initial_theta = gradient_descent(training_data_set, initial_theta[0], initial_theta[1])


final_theta0 = initial_theta[0]
final_theta1 = initial_theta[1]

print(f'theta0 = {final_theta0}\ntheta1 = {final_theta1}')

Output:

When initial_theta = [0, 0]

theta0 = 0.27311526522692103
theta1 = 0.7771301328221445


When initial_theta = [1, 1]

theta0 = 0.8829506006170339
theta1 = 0.6669442287905096

Answer 1

Convergence

You've run five iterations of gradient descent over just 5 training samples with a (probably reasonable) learning rate of 0.01. That is not expected to bring you to a "final" answer of your problem - you'd need to do many iterations of gradient descent just like you implemented, repeating the process until your thetas converge to a stable value. Then it'd make sense to compare the resulting values.

Replace the 5 in for i in range(5) with 5000 and then look at what happens. It might be illustrative to plot the decrease of the error rate / cost function to see how fast the process converges to a solution.

Answer 2

This is not a problem rather a very usual thing. For that you need to understand how gradient decent works. Every time you randomly initialise your parameters the hypothesis starts it's journey from a random place. With every iteration it updates the parameters so that the cost function converges. In your case as you have ran your gradient decent just for 5 iteration, for different initialisation it ends up with too much different results. Try higher iterations you will see significant similarity even with different initialisation. If i could use visualisation that would be helpful for you.

Answer 3

Here is how I see gradient descent: imagine that you are high up on a rocky mountainside in the fog. Because of the fog, you cannot see the fastest path down the mountain. So, you look around your feet and go down based on what you see nearby. After taking a step, you look around your feet again, and take another step. Sometimes this will trap you in a small low spot where you cannot see any way down (a local minimum) and sometimes this will get you safely to the bottom of the mountain (global minimum). Starting from different random locations on the foggy mountainside might trap you in different local minima, though you might find your way down safely if the random starting location is good.