第一梯度下降：如何归一化X和Y？

Question

I'm doing my first gradient descent ever , following a course about Machine Learning.
But it doesn't seem to work correctly as it oscillates (converges then diverges then converges ... ) and at the end , the result is not appreciate.

Maybe it's because I did'nt normalize my X and Y but I don't know how to do it ... I've tried a way with sklearn StandardScaler , but got an error. I don't know what is going wrong.

I'm using Tensorflow 1.3.0 and jupyter.

Here's my code :

    #from sklearn.preprocessing import StandardScaler
    #scaler=StandardScaler()

    n_epochs=1000
    learning_rate=0.01
    X=tf.constant(housing_data_plus_bias,dtype=tf.float32,name="X")
    #X_norm=scaler.fit_transform(X)
    Y=tf.constant(housing.target.reshape(-1,1),dtype=tf.float32,name="Y")
    theta=tf.Variable(tf.random_uniform([n+1,1],-1.0,1.0),name="theta")
    y_pred=tf.matmul(X,theta,name="predictions")  #eq 1.4
    error=y_pred - Y

    mse=tf.reduce_mean(tf.square(error),name="mse") #eq 1.5
    gradients= (2/(m*mse) ) * tf.matmul(tf.transpose(X),error) 
    training_op = tf.assign(theta,theta - learning_rate * gradients)

    init=tf.global_variables_initializer()

    with tf.Session() as sess:
        sess.run(init)
        print("   Y   ")
        print(Y.eval())
        print("   X   ")
        print(X.eval())
        for epoch in range(n_epochs):
            if epoch%100==0:
                print("Epoch",epoch,"MSE =",mse.eval())
            sess.run(training_op)

        best_theta=theta.eval()

and Here's what I get :

Y   
[[4.526]
 [3.585]
 [3.521]
 ...
 [0.923]
 [0.847]
 [0.894]]
   X   
[[   1.           8.3252      41.        ...    2.5555556   37.88
  -122.23     ]
 [   1.           8.3014      21.        ...    2.1098418   37.86
  -122.22     ]
 [   1.           7.2574      52.        ...    2.80226     37.85
  -122.24     ]
 ...
 [   1.           1.7         17.        ...    2.3256352   39.43
  -121.22     ]
 [   1.           1.8672      18.        ...    2.1232092   39.43
  -121.32     ]
 [   1.           2.3886      16.        ...    2.616981    39.37
  -121.24     ]]
Epoch 0 MSE = 511820.7
Epoch 100 MSE = 775760.0
Epoch 200 MSE = 2181710.8
Epoch 300 MSE = 115924.266
Epoch 400 MSE = 7663049.0
Epoch 500 MSE = 2283198.2
Epoch 600 MSE = 586127.75
Epoch 700 MSE = 7143360.5
Epoch 800 MSE = 15567712.0
Epoch 900 MSE = 2333040.0

But what's going wrong? I thought that normalize will only allow to converge faster.

Answer 1

From the looks of your code, you aren't using an optimizer for the Gradient Descent algorithm. I suggest to use an optimizer and then check the MSE again. Theoretically, it should improve. Here is an example of one Gradient Descent optimizer,

n_epochs=1000
learning_rate=0.01
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate) # Play around with learning rates and check the accuracy
X=tf.constant(housing_data_plus_bias,dtype=tf.float32,name="X")
#X_norm=scaler.fit_transform(X)
Y=tf.constant(housing.target.reshape(-1,1),dtype=tf.float32,name="Y")
theta=tf.Variable(tf.random_uniform([n+1,1],-1.0,1.0),name="theta")
y_pred=tf.matmul(X,theta,name="predictions")  #eq 1.4
error=y_pred - Y

mse=tf.reduce_mean(tf.square(error),name="mse") #eq 1.5
training_op = optimizer.minimize(mse)

This is using a built in Optimizer from TensorFlow. You can opt to manually code the optimizer for your gradient descent algorithm.

Here is a link to blog site which explains different optimizers and Gradient descent in detail, http://ruder.io/optimizing-gradient-descent/