多输出回归问题的多重损失

Question

So I am trying to train a CNN model to predict 4 real-valued outputs (regression problem), I tried using a Mean Squared Error as a loss function, My question is if I branch the output layer into 4 different output layers with 4 different losses(4 MSE) does this make the network performs better regarding that the weights for the last layers are updated separately. Thank you

Answer 1

You said it is the same as in the link: https://towardsdatascience.com/deep-neural-networks-for-regression-problems-81321897ca33

So you have something similar to this

# The Output Layer :
NN_model.add(Dense(4, kernel_initializer='normal',activation='linear'))

A FCN with 4 neurones and thus 4 outputs. So your question was if you could instead have 4 output layers. So instead of one layer with 4 neuron you will have 4 layers with one, all in parallel connected to the last FCN layer of the backbone network.

First of all it doesn't change the network architecture. Best to look at this image . You see that every neurons in the last layer is connected to all neurones in the second last. So by splitting the layer up into multiple layers you don't really change the architecture in any way. You still have 4 single neurons connected to all previous ones.

What it will do is change the loss. Instead of calculating the MSE over all outputs, over all batches, you will not calculate the MSE over one output over all batches 4 times. During back propagation the 4 individual gradients, from the 4 layers will be added up. So something like this will happen.

loss_split = mse(output1) + mse(output2) + mse(output3) +mse(output4)

This will compare to the loss with the 4 neuron layer the following

loss = loss_split / num_outputs

In the end this means you change the magnitude of the gradient but not the direction. Instead you could just change the learning rate or multiply the loss by num_outputs .

All in all it will make no sense in splitting the output layer into four in this case. It has no effect on the overall architecture and just complicates the network and introduces unnecessary overhead.

Btw if you really want to change things up regarding the loss and gradients you could try to use a different loss like Smooth-L1 loss. It can be more robust and perform better depending on your data.