使用 numpy 的小批量梯度下降

Question

I'm currently working through chapter four of Hands on Machine Learning with Sci-kit Learn, Keras and Tensorflow and am stuck on trying to implement a mini batch optimization using Numpy.

The cost function is an MSE ( as the example provided is using gradient descent to optimize a linear regression. )

The code follows below:

import numpy as np
X = 2 * np.random.rand(100,1) # Simulate Linear Data
y = 4 + 3 * X + np.random.randn(100,1)
X_b = np.c_[np.ones((100,1)),X]

## Mini batch gradient descent

n_epochs = 50
MINIBATCH_SIZE = 10
rng = np.random.default_rng()
t0,t1 = 5, 50 # learning schedule hyperparameters
m = 100

def learning_schedule(t):
    return t0/(t+t1)

theta = np.random.randn(2,1) # random intitalise weights

mbgd = np.array(
    [[],[],]
)


# 50 epochs, and in each epoch, we are training 100 times, and this can be with the same/different x point ( its random!)
# We carry across the theta/weights across epoch's

for epoch in range(n_epochs): 
    for i in range(m): 
        random_indexes = rng.choice(m,MINIBATCH_SIZE,False)
        xi =  X_b[random_indexes]
        yi = y[random_indexes]
        gradients = 2 * xi.T.dot(xi.dot(theta) - yi) # This is the partial deriv cost function ( MSE)
        eta = learning_schedule(epoch * m + i)
        theta = theta - eta * gradients
        mbgd = np.concatenate([mbgd,theta],axis = 1)

However, when I look at the values of mbgd , it appears to spiral from values such as -2.5871605790804576e+17 and 3.197175730045784e+17 .

I was wondering if I had implemented minibatch gradient descent correctly, as in the book their graph looks much more stable:

Answer 1

The code seems okay. The behavior seems very unstable in the first 100 iterations but it tends to stabilize around the expected values (ie, the linear function parameters).

To check that you can plot mbdg values and look at the values to which they converge. However the plot in the book is a little bit misleading, since the estimated values don't seems to juggle very intensely in the beginning.

I re run the same code using new parameters for the function learning_schedule . I chose t_1 = 500 so that the eta stays at low and thus making the estimated parameters less juggly.

The gradient descent seems less unstable. And since the problem is easy this did not make the computations slow.

import matplotlib.pyplot as plt
plt.figure()
plt.plot(np.arange(5000), mbgd[0,:5000])
plt.plot(np.arange(5000), mbgd[1,:5000])

enter image description here

If I try to plot like the book's figure, you can see that it converges very fast. Here, I plotted only the first 100 iterations.

plt.figure()
plt.plot(mbgd[0,:100], mbgd[1,:100], '-')

enter image description here

Here is another way to look at the optimization result (the first 500 iterations):

from matplotlib import cm
linx = np.arange(0,2,.01)
plt.plot(X_b[:,1], y.squeeze(), 'o')
viridis = cm.get_cmap('viridis', 12)
colmap = viridis(np.linspace(0, 1, 50))
for i in range(50):
    plt.plot(linx, mbgd[0,i*10]+linx*mbgd[1,i*10], color=colmap[i])

enter image description here