在线性回归上使用梯度下降会产生不正确的偏差

Question

I've got a toy example set up of a linear regression model with one input variable and one output variable. The problem that I'm encountering is that the output for the bias is far off from the generated data. If I manually set the bias then it will produce a weight and bias which is close enough to the original.

I've written two pieces of code gen_data which generates data and GradientDescent which performs that gradient descent algorithm to find the weight and bias.

def gen_data(num_points=50, slope=1, bias=10, x_max=50):
    f = lambda z: slope * z + bias
    x = np.zeros(shape=(num_points, 1))
    y = np.zeros(shape=(num_points, 1))

    for i in range(num_points):
        x_temp = np.random.uniform()*x_max
        x[i] = x_temp
        y[i] = f(x_temp) + np.random.normal(scale=3.0)

    return (x, y)

# \mathbb{R}^1 with no regularization
def gradientDescent2(x, y, learning_rate=0.0001, epochs=100):
    theta = np.random.rand()
    bias = np.random.rand()

    for i in range(0, epochs):
        loss = (theta * x + bias) - y
        cost = np.mean(loss**2) / 2
        # print('Iteration {} | Cost: {}'.format(i, cost))

        grad_b = np.mean(loss)
        grad_t = np.mean(loss*x)

        # updates
        bias -= learning_rate * grad_b
        theta -= learning_rate * grad_t

    return (theta, bias)

Answer 1

1. If you want to use batch update, don't set your batch_size equals to your simple size. (I also believe that batch_update is not very suitable for this case.)

2.Your gradient calculation and parameter update are incorrect, the gradient should be:

grad_b =  1
grad_t =  x

For the parameter update, you should always trying to minimize the loss , so it should be

if loss>0:
  bias -= learning_rate * grad_b
  theta -= learning_rate * grad_t
elif loss< 0:
  bias += learning_rate * grad_b
  theta += learning_rate * grad_t

After all, below is the modified code works well. import numpy as np import sys

def gen_data(num_points=500, slope=1, bias=10, x_max=50):
    f = lambda z: slope * z + bias
    x = np.zeros(shape=(num_points))
    y = np.zeros(shape=(num_points))

    for i in range(num_points):
        x_temp = np.random.uniform()*x_max
        x[i] = x_temp
        y[i] = f(x_temp) #+ np.random.normal(scale=3.0)
        #print('x:',x[i],'        y:',y[i])

    return (x, y)

def gradientDescent2(x, y, learning_rate=0.001, epochs=100):
    theta = np.random.rand()
    bias = np.random.rand()

    for i in range(0, epochs):
      for j in range(len(x)):
        loss = (theta * x[j] + bias) - y[j]
        cost = np.mean(loss**2) / 2
        # print('Iteration {} | Cost: {}'.format(i, cost))

        grad_b = 1
        grad_t = x[j]

        if loss>0:
            bias -= learning_rate * grad_b
            theta -= learning_rate * grad_t
        elif loss< 0:
            bias += learning_rate * grad_b
            theta += learning_rate * grad_t

    return (theta, bias)

def main():
    x,y =gen_data()
    ta,bias = gradientDescent2(x,y)
    print('theta:',ta)
    print('bias:',bias)

if __name__ == '__main__':
    sys.exit(int(main() or 0))