Linear regression using Gradient Descent

Question

I'm facing some issues trying to find the linear regression line using Gradient Descent, getting to weird results. Here is the function:

def gradient_descent(m_k, c_k, learning_rate, points):
    n = len(points)
    dm, dc = 0, 0 
    for i in range(n):
        x = points.iloc[i]['alcohol']
        y = points.iloc[i]['total']
        dm += -(2/n) * x * (y - (m_k * x + c_k))  # Partial der in m
        dc += -(2/n) * (y - (m_k * x + c_k))  # Partial der in c
    m = m_k - dm * learning_rate
    c = c_k - dc * learning_rate
    return m, c 

And combined with a for loop

l_rate = 0.0001
m, c = 0, 0
epochs = 1000

for _ in range(epochs):
    m, c = gradient_descent(m, c, l_rate, dataset)

plt.scatter(dataset.alcohol, dataset.total)
plt.plot(list(range(2, 10)), [m * x + c for x in range(2,10)], color='red')
plt.show()

Gives this result:

• Slope: 2.8061974241244196
• Y intercept: 0.5712221080810446

The problem is though that taking advantage of sklearn to compute the slope and intercept, ie

model = LinearRegression(fit_intercept=True).fit(np.array(dataset['alcohol']).copy().reshape(-1, 1),
                                                 np.array(dataset['total']).copy())

I get something completely different:

• Slope: 2.0325063
• Intercept: 5.8577761548263005

Any idea why? Looking on SO I've found out that a possible problem could be a too high learning rate, but as stated above I'm currently using 0.0001

Answer 1

Sklearn's LinearRegression doesn't use gradient descent - it uses Ordinary Least Squares (OLS) Regression which is a non-iterative method .

For your model, you might consider randomly initialising m, c rather than starting with 0,0. You could also consider adjusting the learning rate or using an adaptive learning rate.