InvalidArgumentError：logits 和 labels 必须具有相同的第一维 seq2seq Tensorflow

Question

I am getting this error in seq2seq.sequence_loss even though first dim of logits and labels has same dimension, ie batchSize

I have created a seq2seq model in TF 1.0 version. My loss function is as follows:

    logits  = self.decoder_logits_train
    targets = self.decoder_train_targets
    self.loss     = seq2seq.sequence_loss(logits=logits, targets=targets, weights=self.loss_weights)
    self.train_op = tf.train.AdamOptimizer().minimize(self.loss)

I am getting following error on running my network while training:

InvalidArgumentError (see above for traceback): logits and labels must have the same first dimension, got logits shape [1280,150000] and labels shape [1536]
     [[Node: sequence_loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits = SparseSoftmaxCrossEntropyWithLogits[T=DT_FLOAT, Tlabels=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](sequence_loss/Reshape, sequence_loss/Reshape_1)]]

I confirm the shapes of logits and targets tensors as follows:

a,b = sess.run([model.decoder_logits_train, model.decoder_train_targets], feed_dict)
print(np.shape(a)) # (128, 10, 150000) which is (BatchSize, MaxSeqSize, Vocabsize)
print(np.shape(b)) # (128, 12) which is (BatchSize, Max length of seq including padding)

So, since the first dimension of targets and logits are same then why I am getting this error?

Interestingly, in error u can observe that the dimension of logits is mentioned as (1280, 150000) , which is (128 * 10, 150000) [product of first two dimension, vocab_size] , and same for targets ie (1536) , which is (128*12) , again product of first two dimension?

Note: Tensorflow 1.0 CPU version

Answer 1

The error message seems to be a bit misleading, as you actually need first and second dimensions to be the same. This is written here :

logits: A Tensor of shape [batch_size, sequence_length, num_decoder_symbols] and dtype float. The logits correspond to the prediction across all classes at each timestep.

targets: A Tensor of shape [batch_size, sequence_length] and dtype int. The target represents the true class at each timestep.

This also makes sense, as logits are probability vectors, while targets represent the real output, so they need to be of the same length.

Answer 2

maybe your way of padding wrong. if you padded _EOS to the end of target seq, then the max_length(real length of target sentence) should add 1 to be [batch, max_len+1]. Since you padded _GO and _EOS, your target sentence length should add 2, which makes it equals 12.

I read some other people's implementation of NMT, they only padded _EOS for target sentence, while _GO for input of decoder. Tell me if I'm wrong.

Answer 3

I had the same error as you and I understood the problem:

The problem:

You run the decoder using this parameters:

• targets are the decoder_inputs. They have length max_length because of padding. Shape: [batch_size, max_length]
• sequence_length are the non-padded-lengths of all the targets of your current batch. Shape: [batch_size]

Your logits, that are the output tf.contrib.seq2seq.dynamic_decode has shape:

[batch_size, longer_sequence_in_this_batch, n_classes]

Where longer_sequence_in_this_batch is equal to tf.reduce_max(sequence_length)

So, you have a problem when computing the loss because you try to use both:

• Your logits with 1st dimension shape longer_sequence_in_this_batch
• Your targets with 1st dimension shape max_length

Note that longer_sequence_in_this_batch <= max_length

How to fix it:

You can simply apply some padding to your logits.

logits  = self.decoder_logits_train
targets = self.decoder_train_targets

paddings = [[0, 0], [0, max_length-tf.shape(logits)[1]], [0, 0]]
padded_logits = tf.pad(logits, paddings, 'CONSTANT', constant_values=0)


self.loss = seq2seq.sequence_loss(logits=padded_logits, targets=targets, 
                                  weights=self.loss_weights)

Using this method,you ensure that your logits will be padded as the targets and will have dimension [batch_size, max_length, n_classes]

For more information about the pad function, visit Tensorflow's documentation