为什么我的网络无法学习？

Question

So I created a convolutional network in tensorflow but the accuracy will not change at all. I am trying to get it to tell the difference between triangles and circles. They are different colors and similar sizes. This is the code for the network. Also when I tried with a fully connected network the accuracy was almost 1.

x = tf.placeholder("float", shape=[None, 3072])
y_ = tf.placeholder("float", shape=[None, 2])

W = tf.Variable(tf.zeros([3072, 2]))
b = tf.Variable(tf.zeros([2]))

y = tf.nn.softmax(tf.matmul(x,W) + b)

def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)

def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                           strides=[1, 2, 2, 1], padding='SAME')

W_conv1 = weight_variable([4, 4, 3, 32])
b_conv1 = bias_variable([32])

x_image = tf.reshape(x, [-1,32,32,3])

h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)

W_conv2 = weight_variable([4, 4, 32, 64])
b_conv2 = bias_variable([64])

h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)

W_fc1 = weight_variable([8 * 8 * 64, 1024])
b_fc1 = bias_variable([1024])

h_pool2_flat = tf.reshape(h_pool2, [-1, 8*8*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

keep_prob = tf.placeholder("float")
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

W_fc2 = weight_variable([1024, 2])
b_fc2 = bias_variable([2])

y_conv=tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)

cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv + 1e-9))
train_step = tf.train.AdamOptimizer(1e-6).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

self.feedin = np.reshape(self.inlist, (-1, 1, 32, 32, 3))
print(self.feedin)
sess.run(tf.initialize_all_variables())
for i in range(10000):
    j = i%int(self.ent)
    if i%100 == 0:
        train_accuracy = accuracy.eval(feed_dict={
            x:self.inlist, y_: self.outListm, keep_prob: 1.0})
        print("step %d, training accuracy %g"%(i, train_accuracy))
    train_step.run(feed_dict={x: self.feedin[j], y_: self.outListm[j], keep_prob: 0.5})

These are two of the images.

This is what I used to create self.in. I have changed it so that the shape of the image remains however the problem is still there.

name = QtGui.QFileDialog.getOpenFileNames(self, 'Open File')
fname = [str(each) for each in name]
flist = []
dlist = []
self.inlist = [11, 32, 32, 3]
for n, val in enumerate(name):
    flist.append(val)
    img = Image.open(flist[n])
    img.load()
    data = np.asarray(img, dtype = "float32")
    dlist.append(data)
self.inlist = np.concatenate([arr[np.newaxis] for arr in dlist])

For self out I have a list with 2 elements where the first element is 2 if it is a triangle and the second element is 2 if it is a circle.

Answer 1

You cannot initialize all parameters to zeros(or any constant), which is almost a commonsense for almost all kinds of neural network.

Lets just imagine a simplest feed-forward network with all weight matrices initialized to the same constant(including but not just zero), what's gonna happen? No matter what your input vector is, the activations(outputs) of all neurons in the same layer will be the same! which is definitely not what you want. And in your case you initialize them all to be zeros, which makes it even worse. since besides the downsides above, ReLU is not even derivable at point zero.

So the best practice for you is initializing your weight matrix(W) to random values so as to "break the symmetry". You can just do it by random.randn() , but there are many tricks to do this for even better performance, like Xavier initialiation, MSRA initialization, etc. For ReLU activation function in your case, one thing might guide your selection among all these initialization strategies is that you better initialize your weight matrix be slightly positive, in case that the input of ReLU function would be negative, which might make ReLU units become "dead" ones(gradients being zeros forever).

Answer 2

因为您的1e-6学习率太低，所以每次训练的准确性都会提高得很少。

Answer 3

Like many people say, you cannot initialize weight parameters with zeros. The weights will always update with the same numerical values.

Therefore, we initialize with random values. In other comments, you ask how to do this. This piece is already in your code. Call your function weight_variable to obtain a weight matrix with random initialization. Or, if you want to do it inline

tf.Variable(tf.truncated_normal(shape, stddev=0.1))