[英]PyTorch custom loss function
How should a custom loss function be implemented ?应该如何实现自定义损失函数? Using below code is causing error :
使用以下代码导致错误:
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F
num_epochs = 20
x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])
num_epochs = 200
class cus2(torch.nn.Module):
def __init__(self):
super(cus2,self).__init__()
def forward(self, outputs, labels):
# reshape labels to give a flat vector of length batch_size*seq_len
labels = labels.view(-1)
# mask out 'PAD' tokens
mask = (labels >= 0).float()
# the number of tokens is the sum of elements in mask
num_tokens = int(torch.sum(mask).data[0])
# pick the values corresponding to labels and multiply by mask
outputs = outputs[range(outputs.shape[0]), labels]*mask
# cross entropy loss for all non 'PAD' tokens
return -torch.sum(outputs)/num_tokens
x = torch.tensor([x1,x2,x3,x4]).float()
y = torch.tensor([0,1,1,0]).long()
train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)
device = 'cpu'
input_size = 2
hidden_size = 100
num_classes = 2
learning_rate = .0001
class NeuralNet(nn.Module) :
def __init__(self, input_size, hidden_size, num_classes) :
super(NeuralNet, self).__init__()
self.fc1 = nn.Linear(input_size , hidden_size)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(hidden_size , num_classes)
def forward(self, x) :
out = self.fc1(x)
out = self.relu(out)
out = self.fc2(out)
return out
for i in range(0 , 1) :
model = NeuralNet(input_size, hidden_size, num_classes).to(device)
criterion = nn.CrossEntropyLoss()
# criterion = Regress_Loss()
# criterion = cus2()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
total_step = len(train_loader)
for epoch in range(num_epochs) :
for i,(images , labels) in enumerate(train_loader) :
images = images.reshape(-1 , 2).to(device)
labels = labels.to(device)
outputs = model(images)
loss = criterion(outputs , labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# print(loss)
outputs = model(x)
print(outputs.data.max(1)[1])
makes perfect predictions on training data :对训练数据做出完美的预测:
tensor([0, 1, 1, 0])
Using a custom loss function from here :使用这里的自定义损失函数:
is implemented in above code as cus2
在上面的代码中实现为
cus2
Un-commenting code # criterion = cus2()
to use this loss function returns : # criterion = cus2()
注释代码# criterion = cus2()
以使用此损失函数返回:
tensor([0, 0, 0, 0])
A warning is also returned :还会返回警告:
UserWarning: invalid index of a 0-dim tensor.
用户警告:0 维张量的无效索引。 This will be an error in PyTorch 0.5.
这将是 PyTorch 0.5 中的错误。 Use tensor.item() to convert a 0-dim tensor to a Python number
使用 tensor.item() 将 0-dim 张量转换为 Python 数字
I've not implemented the custom loss function correctly ?我没有正确实现自定义损失函数?
Your loss function is programmatically correct except for below:您的损失函数在编程上是正确的,但以下情况除外:
# the number of tokens is the sum of elements in mask
num_tokens = int(torch.sum(mask).data[0])
When you do torch.sum
it returns a 0-dimensional tensor and hence the warning that it can't be indexed.当你执行
torch.sum
它返回一个 0 维张量,因此警告它不能被索引。 To fix this do int(torch.sum(mask).item())
as suggested or int(torch.sum(mask))
will work too.要解决此问题,请按照建议执行
int(torch.sum(mask).item())
或int(torch.sum(mask))
也可以。
Now, are you trying to emulate the CE loss using the custom loss?现在,您是否尝试使用自定义损失来模拟 CE 损失? If yes, then you are missing the
log_softmax
如果是,那么您缺少
log_softmax
To fix that add outputs = torch.nn.functional.log_softmax(outputs, dim=1)
before statement 4. Note that in case of tutorial that you have attached, log_softmax
is already done in the forward call.要修复在语句 4 之前添加
outputs = torch.nn.functional.log_softmax(outputs, dim=1)
请注意,如果您附加了教程,则log_softmax
已经在前向调用中完成。 You can do that too.你也可以那样做。
Also, I noticed that the learning rate is slow and even with CE loss, results are not consistent.另外,我注意到学习速度很慢,即使有 CE 损失,结果也不一致。 Increasing the learning rate to 1e-3 works well for me in case of custom as well as CE loss.
在自定义和 CE 损失的情况下,将学习率增加到 1e-3 对我来说效果很好。
Here are a few examples of custom loss functions that I came across in this Kaggle Notebook .以下是我在此 Kaggle Notebook 中遇到的一些自定义损失函数示例。 It provides implementations of the following custom loss functions in
PyTorch
as well as TensorFlow
.它在
PyTorch
和TensorFlow
提供了以下自定义损失函数的PyTorch
。
Loss Function Reference for Keras & PyTorch Keras 和 PyTorch 的损失函数参考
I hope this will be helpful for anyone looking to see how to make your own custom loss functions.我希望这对希望了解如何制作自己的自定义损失函数的人有所帮助。
If you use torch functions you should be fine如果您使用手电筒功能,您应该没问题
import torch
def my_custom_loss(output, target):
loss = torch.mean((output-target*2)**3)
return loss
# Forward pass to the Network
# then,
loss.backward()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.