[英]Sparse Categorical CrossEntropy causing NAN loss
So, I've been trying to implement a few custom losses, and so thought I'd start off with implementing SCE loss, without using the built in TF object.所以,我一直在尝试实现一些自定义损失,所以我想我会从实现 SCE 损失开始,而不使用内置的 TF object。 Here's the function I wrote for it.
这是我为它写的 function。
def custom_loss(y_true, y_pred):
print(y_true, y_pred)
return tf.cast(tf.math.multiply(tf.experimental.numpy.log2(y_pred[y_true[0]]), -1), dtype=tf.float32)
y_pred is the set of probabilties, and y_true is the index of the correct one. y_pred 是概率集合,y_true 是正确概率的索引。 This setup should work according to all that I've read, but it returns NAN loss.
这个设置应该根据我读过的所有内容工作,但它会返回 NAN 丢失。
I checked if there's a problem with the training loop, but it works prefectly with the builtin losses.我检查了训练循环是否有问题,但它与内置损失完美配合。
Could someone tell me what the problem is with this code?有人能告诉我这段代码有什么问题吗?
You can replicate the SparseCategoricalCrossentropy()
loss function as follows您可以复制
SparseCategoricalCrossentropy()
损失 function 如下
import tensorflow as tf
def sparse_categorical_crossentropy(y_true, y_pred, clip=True):
y_true = tf.convert_to_tensor(y_true, dtype=tf.int32)
y_pred = tf.convert_to_tensor(y_pred, dtype=tf.float32)
y_true = tf.one_hot(y_true, depth=y_pred.shape[1])
if clip == True:
y_pred = tf.clip_by_value(y_pred, 1e-7, 1 - 1e-7)
return - tf.reduce_mean(tf.math.log(y_pred[y_true == 1]))
Note that the SparseCategoricalCrossentropy()
loss function applies a small offset ( 1e-7
) to the predicted probabilities in order to make sure that the loss values are always finite, see also this question .请注意,
SparseCategoricalCrossentropy()
损失 function 对预测概率应用了一个小的偏移量 ( 1e-7
),以确保损失值始终是有限的,另请参见此问题。
y_true = [1, 2]
y_pred = [[0.05, 0.95, 0.0], [0.1, 0.8, 0.1]]
print(tf.keras.losses.SparseCategoricalCrossentropy()(y_true, y_pred).numpy())
print(sparse_categorical_crossentropy(y_true, y_pred, clip=True).numpy())
print(sparse_categorical_crossentropy(y_true, y_pred, clip=False).numpy())
# 1.1769392
# 1.1769392
# 1.1769392
y_true = [1, 2]
y_pred = [[0.0, 1.0, 0.0], [0.0, 1.0, 0.0]]
print(tf.keras.losses.SparseCategoricalCrossentropy()(y_true, y_pred).numpy())
print(sparse_categorical_crossentropy(y_true, y_pred, clip=True).numpy())
print(sparse_categorical_crossentropy(y_true, y_pred, clip=False).numpy())
# 8.059048
# 8.059048
# inf
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.