PyTorch 中的简单 L1 丢失

Question

I want to calculate L1 loss in a neural network, I came across this example at https://discuss.pytorch.org/t/simple-l2-regularization/139/2 , but there are some errors in this code.

Is this really how to calculate L1 Loss in a NN or is there a simpler way?

l1_crit = nn.L1Loss()
reg_loss = 0
for param in model.parameters():
    reg_loss += l1_crit(param)

factor = 0.0005
loss += factor * reg_loss

Is this equivalent in any way to simple doing:

loss = torch.nn.L1Loss()

I assume not, because I am not passing along any network parameters. Just checking if there isn existing function to do this.

Answer 1

If I am understanding well, you want to compute the L1 loss of your model (as you say in the begining). However I think you might got confused with the discussion in the pytorch forum.

From what I understand, in the Pytorch forums, and the code you posted, the author is trying to normalize the network weights with L1 regularization. So it is trying to enforce that weights values fall in a sensible range (not too big, not too small). That is weights normalization using L1 normalization (that is why it is using model.parameters() ). Normalization takes a value as input and produces a normalized value as output. Check this for weights normalization: https://pytorch.org/docs/master/generated/torch.nn.utils.weight_norm.html

On the other hand, L1 Loss it is just a way to determine how 2 values differ from each other, so the "loss" is just measure of this difference. In the case of L1 Loss this error is computed with the Mean Absolute Error loss = |xy| where x and y are the values to compare. So error compute takes 2 values as input and produces a value as output. Check this for loss computing: https://pytorch.org/docs/master/generated/torch.nn.L1Loss.html

To answer your question: no, the above snippets are not equivalent, since the first is trying to do weights normalization and the second one, you are trying to compute a loss. This would be the loss computing with some context:

sample, target = dataset[i]
target_predicted = model(sample)
loss = torch.nn.L1Loss()
loss_value = loss(target, target_predicted)