Tensorflow neural network prediction is always the same

Question

I have a deep CNN that predicts a label between "0" and "2" for every pixel in a 3d image. I have trained the model on an image where every pixel is labeled "1". Therefore, when testing the model, I believe every prediction should be "1". Instead the model only predicts "0".

Here is the repository for the whole model: https://github.com/dhasl002/Research-DeepLearning .

Since the code is almost 300 lines, I will include only the relevant code below.

 x = tf.placeholder(tf.float32, shape=[None, 7168])
 y_ = tf.placeholder(tf.float32, shape=[None, 7168, 3])

 W_final = weight_variable([7168,7168,3])
 b_final = bias_variable([7168,3])

 #"final" is the result of the many convolutions
 final_conv = tf.tensordot(final, W_final, axes=[[1], [1]]) + b_final

 cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=final_conv))
 train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
 correct_prediction = tf.equal(tf.argmax(final_conv, 2), tf.argmax(y_, 2))
 accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

 #a is a threshold associate with each pixel, b is the label of each pixel
 a = np.zeros((1,7168),dtype = float)
 b = np.zeros((1,7168, 3), dtype = float)

 #this is a little simplified for clarity of reader
 #TRAINING
 for line in inputFile:
   thresh, label = line.strip().split(",")
   a[0][it] = thresh
   b[0][it][label] = 1
 train_step.run(feed_dict={x: a, y_: b, keep_prob: .5})

 #TESTING
 for line in inputFile:
   thresh, label = line.strip().split(",")
   a[0][it] = thresh
   b[0][it][label] = 1
 temp = sess.run(tf.argmax(final_conv,2), feed_dict={x: a})

I believe that "temp" from the last line should hold the correct predictions (7168 labels - one per pixel). Why does "temp" always result in all "0" labels when it is actually trained on images only with "1" labels?

Answer 1

The data that you've provided contains not only 1 labels, but occasional 2 as well (you can skim through the text files or simply print the label values to see this). Not only does it contradict your idea to train a constant function, it also breaks the one-hot encoding and, thus, the whole algorithm.

Here's the excerpt from your script:

a = np.zeros((1,N*M*P),dtype = float)
b = np.zeros((1,N*M*P, 3), dtype = float)
[...]

with tf.Session() as sess:
   sess.run(tf.global_variables_initializer())
   parent = "..."
   with open(parent) as inf1:
     next(inf1)
     for line5 in inf1:
       line1, maxNum = line5.strip().split(",")
       path = "..."
       num = 0
       while num < maxNum:
         it = 0
         with open(path + str(num) + ".txt") as inf:
           next(inf)
           num = num + 1
           for line in inf:
             [...]
             a[0][it] = thresh
             b[0][it][label] = 1
             it = it + 1

Looking at your code, b is supposed to be a one-hot vector. But note that it's zeroed only when the variable is defined. After that it's assigned to 1 at different indices. The later iterations of the while loop update the same b array, hence it ends up containing several 1 in the later rows of the batch. The cross-entropy loss expects a valid probability distribution, hence with your data its output becomes completely meaningless:

Each row labels[i] must be a valid probability distribution.

Summary : the way you do data processing is too complicated and, as a result, error-prone. Try to organize your input files simpler, so that it could be read into a numpy array (or pandas dataframe) and fed to the session.

Answer 2

Since you're using ReLUs, one possibility is that you're suffering from the Dying ReLU Problem; you could fix this by switching to something like leaky ReLUs.

Beyond that, your model is quite deep and complex; in order to make sure it's working properly you might want to scale it back considerably, test it to see if it gives you reasonable results, and then add things back in stages.

In any event, it seems like your model is much too complex for the problem. A model producing a label for each individual pixel should be pretty simple, since a given pixel's label probably only depends on nearby pixels, and probably not in a very complex way.