线性回归，梯度下降不适用于某些值

Question

I have implemented linear regression with gradient descent but it doesn't work when I need to get decreasing function or constant function.

It works for data like

x_train = np.array([30,60,70,100])
y_train= np.array([60,120,145,195])

where I need to get increasing function like here f(x_train) ≈ 2*x_train=y_train or for data like

x_train = np.array([30,60,70,100])
y_train= np.array([30,60,70,100])

where is f(x_train)=x_train=y_train , but it doesn't work for data like this

x_train = np.array([50,100,150,200])
y_train= np.array([150,100,50,0])

decreasing function f(x_train)=200-x_train=y_train or for constant function f(x_train)=100=y_train

x_train = np.array([50,100,150,200])
y_train= np.array([100,100,100,100])

---

import numpy as np
import matplotlib.pyplot as plt

#doesnt work
#x_train = np.array([50,100,150,200])
#y_train= np.array([150,100,50,0])

#work
x_train = np.array([30,60,70,100])
y_train= np.array([60,120,145,195])

def model(x,w,b):
    return x*w+b;

def cost(y,y_hat):
    return np.sum((y-y_hat)**2)

learning_rate=0.000001
def trainning_round(x_train,y_train,w,b,learning_rate):

    y_hat=model(x_train,w,b)
    print(cost(y_train,y_hat))

    w_gradient=-2*x_train.dot(y_train-y_hat)
    b_gradient=-2*np.sum(y_train-y_hat)

    w-=learning_rate*w_gradient
    b-=learning_rate*b_gradient

    return w,b

num_epoch=200
def train(X,Y):

    w=0
    b=0

    #for plt
    ar = np.arange(0, 200, 0.5)
    def f(t):
        return t*w+b

    for i in range(num_epoch):
            w,b=trainning_round(X,Y,w,b,learning_rate)
            plt.plot(ar,f(ar))
            plt.axis([0, 200, 0, 200])

    plt.plot(X, Y, 'ro')

train(x_train,y_train)
plt.show()

I have tried some other algorithms but they didn't work. Thanks in advance

Answer 1

I modified some parts. It is important that you decide appropriate learning_rate and iteration epoch. That's some difficult. So we use a framework like tensorflow.

import numpy as np
import matplotlib.pyplot as plt

#doesnt work
x_train = np.array([50,100,150,200])
y_train= np.array([150,100,50,0])

#work
# x_train = np.array([30,60,70,100])
# y_train= np.array([60,120,145,195])

def model(x,w,b):
    return x*w+b;

def cost(y,y_hat):
    return np.sum((y-y_hat)**2)/y.size

learning_rate=0.0001
def trainning_round(x_train,y_train,w,b,learning_rate):

    y_hat=model(x_train,w,b)
    j = cost(y_train,y_hat)

    # w_gradient=-2*x_train.dot(y_train-y_hat)
    # b_gradient=-2*np.sum(y_train-y_hat)
    w_gradient=x_train.dot(y_hat-y_train) / y_train.size
    b_gradient=np.sum(y_hat-y_train) / y_train.size

    print(w_gradient, b_gradient)

    w=w-learning_rate*w_gradient
    b=b-learning_rate*b_gradient
    print(j, w,b)
    return w,b

num_epoch=200000
def train(X,Y):

    w=2.1
    b=1.5

    #for plt
    ar = np.arange(0, 200, 0.5)

    for i in range(num_epoch):
        w,b=trainning_round(X,Y,w,b,learning_rate)

    plt.plot(ar,model(ar, w, b))
    plt.axis([0, 300, 0, 200])

    plt.plot(X, Y, 'ro')

train(x_train,y_train)
plt.show()

Answer 2

You can use an adaptive learning rate:

def optimal_learning_rate(X,y,W):
    grad = -np.matmul(X.T,y-np.matmul(X,W))/len(y)
    hessian = np.matmul(X.T,X)
    return np.matmul(grad.T,grad)/np.matmul(np.matmul(grad.T,hessian,grad)

This implementation is for a multidimensional X