Why the accuracy of the training model is not changed in the tensorflow code?

Question

I am a novice of the tensorflow and python. I modified a sample tensorflow code by adding one hidden layer with 50 units, but the accuracy result turned to be wrong and it was not changed no matter how many times the model do training. I cannot find any problem with the code. The dataset is MNIST:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data", one_hot = True)

batch_size = 100
n_batch = mnist.train.num_examples // batch_size

x = tf.placeholder(tf.float32, [None, 784])
y = tf.placeholder(tf.float32, [None, 10])


W = tf.Variable(tf.zeros([784, 50]))
b = tf.Variable(tf.zeros([50]))

Wx_plus_b_L1 = tf.matmul(x,W) + b
L1 = tf.nn.relu(Wx_plus_b_L1)

W_2 = tf.Variable(tf.zeros([50, 10]))
b_2 = tf.Variable(tf.zeros([10]))

prediction = tf.nn.softmax(tf.matmul(L1, W_2) + b_2)


loss = tf.reduce_mean(tf.square(y - prediction))
train_step = tf.train.GradientDescentOptimizer(0.2).minimize(loss)


correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(prediction,1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

init = tf.global_variables_initializer()


with tf.Session() as sess:
   sess.run(init)
   for epoch in range(21):
    for batch in range(n_batch):
        batch_xs, batch_ys = mnist.train.next_batch(batch_size)
        sess.run(train_step, feed_dict={x:batch_xs, y:batch_ys})
    acc = sess.run(accuracy, feed_dict = {x:mnist.test.images, y:mnist.test.labels})
    print("Iter:" + str(epoch) + ", Testing Accuray:" + str(acc))

The output always be the same accuracy: Iter:0, Testing Accuray:0.1135 2018-05-31 18:05:21.039188: W tensorflow/core/framework/allocator.cc:101] Allocation of 31360000 exceeds 10% of system memory. Iter:1, Testing Accuray:0.1135 2018-05-31 18:05:22.551525: W tensorflow/core/framework/allocator.cc:101] Allocation of 31360000 exceeds 10% of system memory. Iter:2, Testing Accuray:0.1135 2018-05-31 18:05:24.070686: W tensorflow/core/framework/allocator.cc:101] Allocation of 31360000 exceeds 10% of system memory. Iter:0, Testing Accuray:0.1135 2018-05-31 18:05:21.039188: W tensorflow/core/framework/allocator.cc:101] Allocation of 31360000 exceeds 10% of system memory. Iter:1, Testing Accuray:0.1135 2018-05-31 18:05:22.551525: W tensorflow/core/framework/allocator.cc:101] Allocation of 31360000 exceeds 10% of system memory. Iter:2, Testing Accuray:0.1135 2018-05-31 18:05:24.070686: W tensorflow/core/framework/allocator.cc:101] Allocation of 31360000 exceeds 10% of system memory. What's wrong in this code? Thank you~~

Answer 1

i think it has to do with the graph. accuracy is never updated as the only op you are calling that gets updated change this code to

with tf.Session() as sess:
   sess.run(init)
   for epoch in range(21):
    for batch in range(n_batch):
        batch_xs, batch_ys = mnist.train.next_batch(batch_size)
        sess.run([train_step,accuracy], feed_dict={x:batch_xs, y:batch_ys})
    acc = sess.run(accuracy, feed_dict = {x:mnist.test.images, y:mnist.test.labels})
    print("Iter:" + str(epoch) + ", Testing Accuray:" + str(acc))

Answer 2

The reason is I initialize all weights and bias to zero. If that so, all of the output of the neurons will be the same. The back propagation behavior of all neurons within the same layer is the same - the same gradient, weight update is the same.This is clearly an unacceptable result.

Answer 3

I have had the same problem with Titanic dataset. What helped was to change the learning rate:

optimize = tf.train.AdamOptimizer(learning_rate=0.000001).minimize(mean_loss)

When I changed it from 0.001, the accuracy finally started changing. Before that, I have tried to play with the number of layers , batch size , hidden layer size but nothing helped.