How to calculate different loss for different input in keras model

Question

My model has two inputs and I want to calculate the loss of the two inputs separately because the loss of input 2 has to be multiplied by a weight. Then add up these two losses as the final loss for the model. The structure is somehow like this:

This is my model:

def final_loss(y_true, y_pred):
    loss = x_loss_value.output + y_model.output*weight
    return loss

def mymodel(input_shape): #pooling=max or avg

    img_input1 = Input(shape=(input_shape[0], input_shape[1], input_shape[2], ))
    image_input2 = Input(shape=(input_shape[0], input_shape[1], input_shape[2], )) 

    #for input1 
    x = Conv2D(32, (3, 3), strides=(2, 2))(img_input1)
    x_dense = Dense(2, activation='softmax', name='predictions')(x)
    x_loss_value = my_categorical_crossentropy_layer(x)[input1_y_true, input1_y_pred]
    x_model = Model(inputs=img_input1, outputs=x_loss_value)

    #for input2 
    y = Conv2D(32, (3, 3), strides=(2, 2))(image_input2)
    y_dense = Dense(2, activation='softmax', name='predictions')(y)
    y_loss_value = my_categorical_crossentropy_layer(y)[input2_y_true, input2_y_pred]
    y_model = Model(inputs=img_input2, outputs=y_loss_value)

    concat = concatenate([x_model.output, y_model.output])
    final_dense = Dense(2, activation='softmax')(concat)

    # Create model.
    model = Model(inputs=[img_input1,image_input2], output = final_dense)
    return model

    model.compile(optimizer = optimizers.adam(lr=1e-7), loss = final_loss, metrics = ['accuracy'])

Most of the related solutions I found just customize the final loss and change the loss in Model.complie(loss=customize_loss) .

However, I need to apply different losses for different inputs. I'm trying to use a customized layer like this, and get my loss value for final the loss calculation:

class my_categorical_crossentropy_layer1(Layer):

    def __init__(self, **kwargs):
        self.is_placeholder = True
        super(my_categorical_crossentropy_layer1, self).__init__(**kwargs)

    def my_categorical_crossentropy_loss(self, y_true, y_pred):
        y_pred = K.constant(y_pred) if not K.is_tensor(y_pred) else y_pred
        y_true = K.cast(y_true, y_pred.dtype)
        return K.categorical_crossentropy(y_true, y_pred, from_logits=from_logits)

    def call(self, y_true, y_pred):
        loss = self.my_categorical_crossentropy_loss(y_true, y_pred)
        self.add_loss(loss, inputs=(y_true, y_pred))
        return loss

But, inside the keras model, I can't figure out how to get the y_true and y_pred of the current epoch/batch for my loss layer. So I can't add x = my_categorical_crossentropy_layer()[y_true, y_pred] to my model.

Is there any way to do the variable calculation like this in the keras model?

Further, can Keras get the previous epoch's training loss or val loss during training process? I want to apply the previous epoch's training loss as my weight in the final loss.

Answer 1

this is my proposal...

your it's a double binary classification problem that you want to carry out using a single fit. the first thing to notice is that you need to take care of dimensionality: your input is 4d while your target is 2d one-hot encoded so your network needs something to reduce dimensionality, for example, flatten or global pooling. after this, you can start fitting creating a single model with two inputs and two outputs and use two losses. in your case, the losses are weighted categorical_crossentropy . keras enable by default to set the loss weights using loss_weights parameters. to reproduce the formula loss1*1+loss2*W set the weights to [1, W] . you can use the loss_weights parameter also specifying different losses for your output in this way losses=[loss1, loss2, ....] which are linearly combined with the weights specified in the loss_weights

below a working example

input_shape = (28,28,3)
n_sample = 10

# create dummy data
X1 = np.random.uniform(0,1, (n_sample,)+input_shape) # 4d
X2 = np.random.uniform(0,1, (n_sample,)+input_shape) # 4d
y1 = tf.keras.utils.to_categorical(np.random.randint(0,2, n_sample)) # 2d
y2 = tf.keras.utils.to_categorical(np.random.randint(0,2, n_sample)) # 2d

def mymodel(input_shape, weight):

    img_input1 = Input(shape=(input_shape[0], input_shape[1], input_shape[2], ))
    img_input2 = Input(shape=(input_shape[0], input_shape[1], input_shape[2], )) 

    # for input1 
    x = Conv2D(32, (3, 3), strides=(2, 2))(img_input1)
    x = GlobalMaxPool2D()(x) # pass from 4d to 2d
    x = Dense(2, activation='softmax', name='predictions1')(x)

    # for input2 
    y = Conv2D(32, (3, 3), strides=(2, 2))(img_input2)
    y = GlobalMaxPool2D()(y) # pass from 4d to 2d
    y = Dense(2, activation='softmax', name='predictions2')(y)

    # Create model
    model = Model([img_input1,img_input2], [x,y])
    model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'],
                  loss_weights=[1,weight])
    
    return model

weight = 0.3
model = mymodel(input_shape, weight)
model.summary()

model.fit([X1,X2], [y1,y2], epochs=2)