在 TF 2.0 中使用 Keras 自定义损失函数

Question

Custom Loss Function in Tensorflow 2.0 Alpha from TF 1.13

I'm trying to use the roc_auc loss function from this library in the model.compile() of TF 2.0. While my implementation does not though errors the loss and accuracies don't move.

I have first converted the 1.0 TF code to 2.0 using the code suggested by Google.

Then I imported the function form the library and used in the following way:

model.compile(optimizer='adam',
              loss=roc_auc_loss,
              metrics=['accuracy',acc0, acc1, acc2, acc3, acc4])

Epoch 17/100
100/100 [==============================] - 20s 197ms/step - loss: 469.7043 - accuracy: 0.0000e+00 - acc0: 0.0000e+00 - acc1: 0.0000e+00 - acc2: 0.0000e+00 - acc3: 0.0000e+00 - acc4: 0.0000e+00 - val_loss: 152.2152 - val_accuracy: 0.0000e+00 - val_acc0: 0.0000e+00 - val_acc1: 0.0000e+00 - val_acc2: 0.0000e+00 - val_acc3: 0.0000e+00 - val_acc4: 0.0000e+00
Epoch 18/100
100/100 [==============================] - 20s 198ms/step - loss: 472.0472 - accuracy: 0.0000e+00 - acc0: 0.0000e+00 - acc1: 0.0000e+00 - acc2: 0.0000e+00 - acc3: 0.0000e+00 - acc4: 0.0000e+00 - val_loss: 152.2152 - val_accuracy: 0.0000e+00 - val_acc0: 0.0000e+00 - val_acc1: 0.0000e+00 - val_acc2: 0.0000e+00 - val_acc3: 0.0000e+00 - val_acc4: 0.0000e+00
Epoch 19/100
 78/100 [======================>.......] - ETA: 4s - loss: 467.4657 - accuracy: 0.0000e+00 - acc0: 0.0000e+00 - acc1: 0.0000e+00 - acc2: 0.0000e+00 - acc3: 0.0000e+00 - acc4: 0.0000e+00

I would like to understand what is wrong with Keras in TF 2.0 its obviously not backpropagating. Thanks.

Answer 1

@ruben Can you share a standalone code to reproduce the issue? I think we need to check function definitions. Did you add @tf.function() on top of the function definitions? Thanks!

Please check the following example (simple example from TF website)

!pip install tensorflow==2.0.0-beta1

import tensorflow as tf
from tensorflow import keras 
import keras.backend as K

# load mnist data
mnist=tf.keras.datasets.mnist
(x_train,y_train),(x_test,y_test)=mnist.load_data()
x_train,x_test=x_train/255.0,x_test/255.0

# Custom Metric1 (for example)
@tf.function()
def customMetric1(yTrue,yPred):
    return tf.reduce_mean(yTrue-yPred)

# Custom Metric2 (for example)
@tf.function()
def customMetric2(yTrue, yPred):
  return tf.reduce_mean(tf.square(tf.subtract(yTrue,yPred)))

model=tf.keras.models.Sequential([
                                  tf.keras.layers.Flatten(input_shape=(28,28)),
                                  tf.keras.layers.Dense(128, activation='relu'),
                                  tf.keras.layers.Dropout(0.2),
                                  tf.keras.layers.Dense(10,activation='softmax')
])

# Compile the model with custom loss functions
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy', customMetric1, customMetric2])

# Fit and evaluate model
model.fit(x_train,y_train,epochs=5)
model.evaluate(x_test,y_test)

Output

WARNING: Logging before flag parsing goes to stderr. W0711 23:57:16.453042 139687207184256 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where Train on 60000 samples

Epoch 1/5 60000/60000 [==============================] - 5s 87us/sample - loss: 0.2983 - accuracy: 0.9133 - customMetric1: 4.3539 - customMetric2: 27.3769 Epoch 2/5 60000/60000 [==============================] - 5s 83us/sample - loss: 0.1456 - accuracy: 0.9555 - customMetric1: 4.3539 - customMetric2: 27.3860

Epoch 3/5 60000/60000 [==============================] - 5s 82us/sample - loss: 0.1095 - accuracy: 0.9663 - customMetric1: 4.3539 - customMetric2: 27.3881

Epoch 4/5 60000/60000 [==============================] - 5s 83us/sample - loss: 0.0891 - accuracy: 0.9717 - customMetric1: 4.3539 - customMetric2: 27.3893

Epoch 5/5 60000/60000 [==============================] - 5s 87us/sample - loss: 0.0745 - accuracy: 0.9765 - customMetric1: 4.3539 - customMetric2: 27.3901

10000/10000 [==============================] - 0s 46us/sample - loss: 0.0764 - accuracy: 0.9775 - customMetric1: 4.3429 - customMetric2: 27.3301 [0.07644735965565778, 0.9775, 4.342905, 27.330126]

Edit 1

Say for example, if you want to use customMetric1 as a custom loss function, then change couple of things as follows and run the code. Hope this helps.

# Custom Metric1 (for example)
@tf.function()
def customMetric1(yTrue,yPred,name='CustomLoss1'):
  yTrue = tf.dtypes.cast(yTrue, tf.float32)
  return tf.reduce_mean(yTrue-yPred)

model.compile(optimizer='adam',loss=customMetric1, metrics=['accuracy', customMetric2])