Keras Categorical_crossentropy 损失实现

Question

I'm trying to reimplement the Categorical Cross Entropy loss from Keras so that I can customize it. I got the following

def CustomCrossEntropy(output, target, axis=-1):
    target = ops.convert_to_tensor_v2_with_dispatch(target)
    output = ops.convert_to_tensor_v2_with_dispatch(output)
    target.shape.assert_is_compatible_with(output.shape)
    # scale preds so that the class probas of each sample sum to 1
    output = output / math_ops.reduce_sum(output, axis, True)
    # Compute cross entropy from probabilities.
    epsilon_ = _constant_to_tensor(epsilon(), output.dtype.base_dtype)
    output = clip_ops.clip_by_value(output, epsilon_, 1. - epsilon_)
    return -math_ops.reduce_sum(target * math_ops.log(output), axis)

It produces different results than the internal function which confuses me, as I just copied the code from github so far. What am I missing here?

Prove:

y_true = [[0., 1., 0.], [0., 0., 1.]]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]
loss = tf.keras.losses.categorical_crossentropy(y_true, y_pred)
customLoss = CustomCrossEntropy(y_true, y_pred)
assert loss.shape == (2,)
print(loss)
print(customLoss)
>>tf.Tensor([0.05129331 2.3025851 ], shape=(2,), dtype=float32)
>>tf.Tensor([ 0.8059049 14.506287 ], shape=(2,), dtype=float32)

Answer 1

You have inverted the arguments of the function in your definition of CustomCrossEntropy , if you double check the source code in GitHub you will see that the first argument is target and the second one is output . If you switch them back you will get the same results as expected.

import tensorflow as tf
from tensorflow.keras.backend import categorical_crossentropy as CustomCrossEntropy

y_true = [[0., 1., 0.], [0., 0., 1.]]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]

y_true = tf.convert_to_tensor(y_true)
y_pred = tf.convert_to_tensor(y_pred)

loss = tf.keras.losses.categorical_crossentropy(y_true, y_pred)
print(loss)
# tf.Tensor([0.05129331 2.3025851 ], shape=(2,), dtype=float32)

loss = CustomCrossEntropy(y_true, y_pred)
print(loss)
# tf.Tensor([0.05129331 2.3025851 ], shape=(2,), dtype=float32)

loss = CustomCrossEntropy(y_pred, y_true)
print(loss)
# tf.Tensor([ 0.8059049 14.506287 ], shape=(2,), dtype=float32)