Keras multiple outputs, customed loss function

Question

I was trying to build a model with two inputs and two outputs. The structure of the model is like below. And I would like to construct a customer loss function with two parts: the difference between 'd_flat' and 't_flat', and the categorical crossentropy loss of layer 'perdict'. The model is like this:

initial_input_domain=tf.keras.Input(shape=(36,36,3))
initial_input_target=tf.keras.Input(shape=(36,36,3))

vgg_base=tf.keras.applications.VGG19(include_top=False,#weights='imagenet',
                                     input_shape=(36,36,3))

domain1=vgg_base(initial_input_domain)
target1=vgg_base(initial_input_target)

d_flat = tf.keras.layers.Flatten(name='d_flat')(domain1)
predictions=tf.keras.layers.Dense(num_classes,name='predict', activation='sigmoid')(d_flat)

t_flat = tf.keras.layers.Flatten(name='t_flat')(target1)
predictions_t=tf.keras.layers.Dense(num_classes,name='predict_t', activation='sigmoid')(t_flat)

fin_model=tf.keras.Model(inputs=[initial_input_domain,initial_input_target], outputs=[predictions, predictions_t])

The loss function I wrote is like this:

def Total_loss(d_flat, t_flat):

    def loss_function(y_true, y_pred):

        Dist_LOSS = 'something does not matter' # the difference of two layers
        loss = K.categorical_crossentropy(y_true,y_pred) + Dist_LOSS
        return loss

    return loss_function

So my question is what is y_pred and y_true in this function? I only want this function to calculate the categorical crossentropy loss of 'predict', which is the left part. What should I do to make keras not calculate the categorical crossentropy loss of the right part? It seems like y_pred and y_true is the combination of the left and right branches. (The label I used for the right part is the correct label, I used for the right part is all 0, which means nothing)

Keras generates these outputs,

Epoch 1/100
6912/6912 [==============================] - 24s 3ms/sample - loss: 0.0315 - predict_loss: 0.0270 - predict_t_loss: 0.0045 - predict_categorical_accuracy: 0.9931 - predict_t_categorical_accuracy: 0.6413

It seems like the loss = predict_loss + predict_t_loss. It should be any predict_t_loss. Any suggestion is appreciated. Thanks!

Answer 1

Custom loss functions can only work with (y_true, y_pred) . If you want to work with other variables that are defined before the final layer(s), like eg d_flat, t_flat , or only part of the output, you have to use model.add_loss . As you can see in the API, you can either define it in your own custom layer (gives you more specific control) or on the model itself. In your case you can do something like this:

model.add_loss(d_flat - f_flat + K.categorical_crossentropy(predictions,y_pred_for_left_output)

where y_pred_for_left_output is the label tensor for this output node. This way you have defined the loss as the difference between the flat layers and the CE of the left output node only. You can adapt this to your specific requirements but this would be the correct approach.