Tensorflow线性回归未收敛至正确成本

Question

I am trying to implement multivariate linear regression in tensorflow (using the Boston Housing Dataset), but it seems like my cost function is converging and the wrong value (24000 in my case). I tried scaling the features, but it still hasn't worked. Any ideas as to what I'm doing wrong? Here is the code:

from sklearn.datasets import load_boston
import numpy as np
import pandas as pd
import tensorflow as tf
from sklearn.cross_validation import train_test_split
import matplotlib.pyplot as plt
from sklearn.metrics import r2_score
from sklearn.preprocessing import MinMaxScaler

rate = 0.000000011
epochs = 100
errors = []

def load_data():
    boston = load_boston()

    bos = pd.DataFrame(boston.data)

    output = pd.DataFrame(boston.target)

    return [bos, output]

xS, yS = load_data()

m = len(yS)

x_train, x_test, y_train, y_test = train_test_split(xS, yS, test_size=0.2)

scaler = MinMaxScaler()
scaler.fit(x_train)
x_train = scaler.transform(x_train)
x_test = scaler.transform(x_test)

theta = tf.Variable(tf.zeros([len(xS.columns), 1]))
X = tf.placeholder(tf.float32, shape=[m, len(xS.columns)])
y = tf.placeholder(tf.float32, shape=[m, 1])
b = tf.Variable(tf.zeros([m, 1]))

model = tf.matmul(tf.transpose(theta), tf.transpose(X)) + b

cost = tf.reduce_sum(tf.square(y-model))/(2*m)

optimizer = tf.train.GradientDescentOptimizer(rate).minimize(cost)

init = [tf.global_variables_initializer(), tf.local_variables_initializer()]

with tf.Session() as sess:
    sess.run(init)
    for e in range(epochs):
        sess.run(optimizer, feed_dict={X:xS, y:yS})
        loss = sess.run(cost, feed_dict={X:xS, y:yS})
        print("cost at step", e, loss)
        errors.append(loss)
        if errors[len(errors)-1] > errors[len(errors)-2]:
            break

    theta_temp = np.array(sess.run(theta))
    b_temp = np.array(sess.run(b))

plt.plot(list(range(len(errors))), errors)
plt.show()
h = np.transpose(np.add(np.matmul(np.transpose(theta_temp), np.transpose(xS)), np.transpose(b_temp)))
print(r2_score(h, yS))

Answer 1

You are doing most of the thing correctly. I will suggest the following changes in your code.

model = tf.matmul(X, theta) + b

Try this with a learning rate 0.001 and epoch 1000 and please report the result.

In your case where you are doing

model = tf.matmul(tf.transpose(theta), tf.transpose(X)) + b

you are making a mistake. First part of the right side is of size (1, m) and the second part is of size (m, 1). Then you are getting a some result because of broadcasting that you are not expecting. That is why you are seeing very bad results with a learning rate of 0.01 or 0.1.

My second suggestion is remove the break criterion.

if errors[len(errors)-1] > errors[len(errors)-2]: break

Stochastic gradients are noisy. There is no proof that you will always reduce the cost if you go into the less gradient direction (may be it is true for this convex problem but I have to think).

from sklearn.datasets import load_boston
import numpy as np
import pandas as pd
import tensorflow as tf
from sklearn.cross_validation import train_test_split
import matplotlib.pyplot as plt
from sklearn.metrics import r2_score
from sklearn.preprocessing import MinMaxScaler

rate = 0.1
epochs = 100
errors = []

def load_data():
    boston = load_boston()

    bos = pd.DataFrame(boston.data)

    output = pd.DataFrame(boston.target)

    return [bos, output]

xS, yS = load_data()


x_train, x_test, y_train, y_test = train_test_split(xS, yS, test_size=0.2)
m = len(y_train)

scaler = MinMaxScaler()
scaler.fit(x_train)
x_train = scaler.transform(x_train)
x_test = scaler.transform(x_test)

theta = tf.Variable(tf.zeros([len(xS.columns), 1]))
X = tf.placeholder(tf.float32, shape=[m, len(xS.columns)])
y = tf.placeholder(tf.float32, shape=[m, 1])
b = tf.Variable(tf.zeros([1]))

model = tf.matmul(X, theta) + b

cost = tf.reduce_sum(tf.square(y-model))/(2*m)

optimizer = tf.train.GradientDescentOptimizer(rate).minimize(cost)

init = [tf.global_variables_initializer(), tf.local_variables_initializer()]

with tf.Session() as sess:
    sess.run(init)
    for e in range(epochs):
        sess.run(optimizer, feed_dict={X:x_train, y:y_train})
        loss = sess.run(cost, feed_dict={X:x_train, y:y_train})
        print("cost at step", e, loss)
        errors.append(loss)