如何修复TensorFlow线性回归在MSE中没有变化？

Question

I'm working on a simple linear regression model to predict the next step in a series. I'm giving it x/y coordinate data and I want the regressor to predict where the next point on the plot will lie.

I'm using dense layers with AdamOptmizer and have my loss function set to:

tf.reduce_mean(tf.square(layer_out - y))

I'm trying to create linear regression models from scratch (I don't want to utilize the TF estimator package here).

I've seen ways to do it by manually specifying weights and biases, but nothing goes into deep regression.

    X = tf.placeholder(tf.float32, [None, self.data_class.batch_size, self.inputs])
    y = tf.placeholder(tf.float32, [None, self.data_class.batch_size, self.outputs])
    layer_input = tf.layers.dense(inputs=X, units=10, activation=tf.nn.relu)
    layer_hidden = tf.layers.dense(inputs=layer_input, units=10, activation=tf.nn.relu)
    layer_out = tf.layers.dense(inputs=layer_hidden, units=1, activation=tf.nn.relu)
    cost = tf.reduce_mean(tf.square(layer_out - y))
    optmizer = tf.train.AdamOptimizer(learning_rate=self.learning_rate)
    training_op = optmizer.minimize(cost)

    init = tf.initialize_all_variables()
    iterations = 10000
    with tf.Session() as sess:
        init.run()
        for iteration in range(iterations):
            X_batch, y_batch = self.data_class.get_data_batch()
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
            if iteration % 100 == 0:
                mse = cost.eval(feed_dict={X:X_batch, y:y_batch})
                print(mse)
        array = []
        for i in range(len(self.data_class.dates), (len(self.data_class.dates)+self.data_class.batch_size)):
            array.append(i)
        x_pred = np.array(array).reshape(1, self.data_class.batch_size, 1)
        y_pred = sess.run(layer_out, feed_dict={X: x_pred})
        print(y_pred)
        predicted = np.array(y_pred).reshape(self.data_class.batch_size)
        predicted = np.insert(predicted, 0, self.data_class.prices[0], axis=0)
        plt.plot(self.data_class.dates, self.data_class.prices)
        array = [self.data_class.dates[0]]
        for i in range(len(self.data_class.dates), (len(self.data_class.dates)+self.data_class.batch_size)):
            array.append(i)

        plt.plot(array, predicted)
        plt.show()

When I run training I'm getting the same loss value over and over again.

It's not being reduced, like it should, why?

Answer 1

The issue is that I'm applying an activation to the output layer. This is causing that output to go to whatever it activates to.

By specifying in the last layer that activation=None the deep regression works as intended.

Here is the updated architecture:

layer_input = tf.layers.dense(inputs=X, units=150, activation=tf.nn.relu)
    layer_hidden = tf.layers.dense(inputs=layer_input, units=100, activation=tf.nn.relu)
    layer_out = tf.layers.dense(inputs=layer_hidden, units=1, activation=None)