Rank mismatch: Rank of labels (received 2) should equal rank of logits minus 1 (received 2)

Question

I'm building DNN to predict if the object is present in the image or not. My network has two hidden layers and the last layer looks like this:

  # Output layer
  W_fc2 = weight_variable([2048, 1])
  b_fc2 = bias_variable([1])

  y = tf.matmul(h_fc1, W_fc2) + b_fc2

Then I have placeholder for labels:

y_ = tf.placeholder(tf.float32, [None, 1], 'Output')

I run training in batches (therefore first argument in Output layer shape is None).

I use the following loss function:

cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(
    y[:, :1], y_[:, :1], name='xentropy')
loss = tf.reduce_mean(cross_entropy, name='xentropy_mean')
predict_hand = tf.greater(y, 0.5)
correct_prediction = tf.equal(tf.to_float(predict_hand), y_)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

But in runtime I got the following error:

Rank mismatch: Rank of labels (received 2) should equal rank of logits minus 1 (received 2).

I guess I should reshape labels layer, but not sure what it expects. I looked up in documentation and it says:

logits: Unscaled log probabilities of rank r and shape [d_0, d_1, ..., d_{r-2}, num_classes] and dtype float32 or float64. labels: Tensor of shape [d_0, d_1, ..., d_{r-2}] and dtype int32 or int64. Each entry in labels must be an index in [0, num_classes).

If I have just single class, what my labels should look like (now it is just 0 or 1)? Any help appreciated

Answer 1

From the documentation* for tf.nn.sparse_softmax_cross_entropy_with_logits :

"A common use case is to have logits of shape [batch_size, num_classes] and labels of shape [batch_size]. But higher dimensions are supported."

So I suppose your labels tensor should be of shape [None] . Note that a given tensor with shape [None, 1] or shape [None] will contain the same number of elements.

Example input with concrete dummy values:

>>> logits = np.array([[11, 22], [33, 44], [55, 66]])
>>> labels = np.array([1, 0, 1])

Where there's 3 examples in the mini-batch, the logits for the first example are 11 and 22 and there's 2 classes: 0 and 1.

* https://www.tensorflow.org/versions/r0.11/api_docs/python/nn.html#sparse_softmax_cross_entropy_with_logits

Answer 2

The problem may be the activation function in your network. Use tf.nn.softmax_cross_entropy_with_logits instead of sparse_softmax. This will solve the issue.

Answer 3

In short, here is implements of it

    cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
            logits=hypothesis,labels=tf.argmax(Y,1)))

sparse_softmax_cross_entropy_with_logits

Computes sparse softmax cross entropy between logits and labels .

Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).

For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both.

NOTE: For this operation, the probability of a given label is considered exclusive . That is, soft classes are not allowed , and the labels vector must provide a single specific index for the true class for each row of logits (each minibatch entry).

For soft softmax classification with a probability distribution for each entry, see softmax_cross_entropy_with_logits .

WARNING: This op expects unscaled logits, since it performs a softmax on logits internally for efficiency. Do not call this op with the output of softmax, as it will produce incorrect results.

A common use case is to have logits of shape [batch_size, num_classes] and labels of shape [batch_size]. But higher dimensions are supported.

Note that to avoid confusion, it is required to pass only named arguments to this function.

softmax_cross_entropy_with_logits_v2 and softmax_cross_entropy_with_logits

Computes softmax cross entropy between logits and labels. (deprecated)

THIS FUNCTION IS DEPRECATED. It will be removed in a future version.

Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow into the labels input on backprop by default. Backpropagation will happen only into logits. To calculate a cross entropy loss that allows backpropagation into both logits and labels, see softmax_cross_entropy_with_logits_v2

Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).

For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both.

here is the same implements of softmax_cross_entropy_with_logits_v2

    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(
            logits=hypothesis,labels=Y))

Answer 4

Why should

"A common use case is to have logits of shape [batch_size, num_classes] and labels of shape [batch_size]. But higher dimensions are supported."

In many tutorials, including here and here , the labels have size [None,10] and the logits have size [None,10] as well.