如果模型 output 层有多个神经元并且只有一个值要预测，那么 Tensorflow model 如何计算损失？

Question

I recently noticed, that I can have and use a Tensorflow model, that has any number of outputs even if the predicted value is just a single real value. For example, I can have 16 outputs from the model and only 1 real target value, however, the model does still train normally with no errors.

My question is how does Tensorflow handle the incorrect size of the output when comparing the output to the target value. How can it still calculate loss? Does it try to get every output as close to the target value as possible, or does it do some kind of averaging?

Answer 1

We use Categorical cross-entropy loss for a model which has any number of outputs and a single prediction value.

Categorical cross-entropy loss = Softmax Activation + Cross-entropy loss

Lets assume we have 4 classes in our model, the softmax activation function will compute the probabilities for 4 classes. The Cross-entropy loss is calculated on these probabilities for the true label class.

Consider the following Example:

The 4 output nodes are Rabbit, Cat, Dog, Squirrel.

True Label: Rabbit

Predictions: Rabbit = 8, Cat = 1, Dog = 4, Squirrel = 2

Softmax Calculation: R = 2980.95/3045.63, C = 2.71/3045.63, D = 54.59/3045.63, S = 7.38/3045.63 (Here, 3045.63 = 2980.95 + 2.71 + 54.59 + 7.38 = 3045.63)

Cross- entropy loss: -(1 * ln(R) + 0 * ln(C) + 0 * ln(D) + 0 * ln(S)) = -(-0.0214 + 0 + 0 + 0) = 0.0214

This is how the model handles the outputs and calculate the loss. Thanks!