[英]Get the Cross Entropy Loss in pytorch as in Keras
I am struggling to port a classification model form keras to pytorch.我正在努力将分类 model 从 keras 移植到 pytorch。 Especially the cross entropy loss seems to return totally different numbers.特别是交叉熵损失似乎返回完全不同的数字。
import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K
y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])
print("Keras", K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred)))
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).argsort(dim=-1).float(), t.tensor(y_true).argmax(dim=-1)))```
prints:印刷:
Keras tf.Tensor([2.3369865], shape=(1,), dtype=float32) Keras tf.Tensor([2.3369865], shape=(1,), dtype=float32)
PyTorch tensor(1.4587) PyTorch 张量(1.4587)
Since I have a custom loss function where cross entropy is a part of it, I would need to get similar if not the same numbers.由于我有一个自定义损失 function ,其中交叉熵是其中的一部分,如果不是相同的数字,我需要得到相似的数字。
The problem is that they have different implementations.问题是它们有不同的实现。
As pytorch docs says, nn.CrossEntropyLoss
combines nn.LogSoftmax()
and nn.NLLLoss()
in one single class.正如 pytorch文档所说, nn.CrossEntropyLoss
将nn.LogSoftmax()
和nn.NLLLoss()
组合在一个 class 中。 However, tensorflow docs specifies that keras.backend.categorical_crossentropy
do not apply Softmax by default unless you set from_logits
is True.但是,tensorflow 文档指定keras.backend.categorical_crossentropy
默认不应用 Softmax,除非您将from_logits
设置为 True。 For this reason, you should not use keras.backend.categorical_crossentropy
without having previously apply softmax unless you use from_logits=True
.出于这个原因,除非您使用from_logits=True
,否则您不应该使用keras.backend.categorical_crossentropy
之前未应用 softmax 。
If you don't want to apply softmax beforehand you should use:如果您不想事先应用 softmax,您应该使用:
import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K
y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])
print("Keras", K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred), from_logits=True))
# output: Keras tf.Tensor([2.408051], shape=(1,), dtype=float32)
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).float(), t.tensor(y_true).argmax(dim=-1)))
# output: PyTorch tensor(2.4081)
Otherwise, you can apply Softmax manually before computing categorical_crossentropy否则,您可以在计算 categorical_crossentropy 之前手动应用 Softmax
import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K
y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])
print("Keras", K.categorical_crossentropy(K.constant(y_true), K.softmax(K.constant(y_pred))))
# output: Keras tf.Tensor([2.408051], shape=(1,), dtype=float32)
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).float(), t.tensor(y_true).argmax(dim=-1)))
# output: PyTorch tensor(2.4081)
So you should not use keras.backend.categorical_crossentropy
with from_logits=False
as you were doing in your example.因此,您不应像在示例中那样使用keras.backend.categorical_crossentropy
和from_logits=False
。
tf.keras.backend.categorical_crossentropy tf.keras.backend.categorical_crossentropy
target : A tensor of the same shape as output. target :与 output 形状相同的张量。
output : A tensor resulting from a softmax (unless from_logits is True, in which case output is expected to be the logits). output :由 softmax 产生的张量(除非 from_logits 为 True,在这种情况下 output 预计是 logits)。
from_logits : Boolean, whether output is the result of a softmax, or is a tensor of logits. from_logits : Boolean,output 是 softmax 的结果,还是 logits 的张量。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.