Keras Masking Output Layer

Question

I have the following Model in Keras:

main_input = Input(shape=(None, 2, 100, 100), dtype='float32', name='input')

hidden = ConvLSTM2D(filters=16, 
                    kernel_size=(5, 5),  
                    padding='same', 
                    return_sequences=False, 
                    data_format='channels_first')(main_input)

output = Conv2D(filters=1, 
                kernel_size=(1, 1), 
                padding='same',
                activation='sigmoid',
                kernel_initializer='glorot_uniform',
                data_format='channels_first',
                name='output')(hidden)

sgd = SGD(lr=0.002, momentum=0.0, decay=0.0, nesterov=False)

I want to multiply the output, which is a 2d array, by a mask (there is a separate mask for each example). How can I do this in Keras?

Answer 1

I think you should input the mask of each sample to the model at the same time.

Here is the suggested code:

from keras.layers import Multiply

main_input = Input(shape=(None, 2, 100, 100), dtype='float32', name='input')

mask=Input(shape=(1, 100, 100), dtype='float32', name='mask')

hidden = ConvLSTM2D(filters=16, 
                    kernel_size=(5, 5),  
                    padding='same', 
                    return_sequences=False, 
                    data_format='channels_first')(main_input)

output = Conv2D(filters=1, 
                kernel_size=(1, 1), 
                padding='same',
                activation='sigmoid',
                kernel_initializer='glorot_uniform',
                data_format='channels_first',
                name='output')(hidden)
output_with_mask=Multiply()([output, mask])

model=Model([main_input, mask], output_with_mask)

The summary is as follow:

    __________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input (InputLayer)              (None, None, 2, 100, 0                                            
__________________________________________________________________________________________________
conv_lst_m2d_7 (ConvLSTM2D)     (None, 16, 100, 100) 28864       input[0][0]                      
__________________________________________________________________________________________________
output (Conv2D)                 (None, 1, 100, 100)  17          conv_lst_m2d_7[0][0]             
__________________________________________________________________________________________________
mask (InputLayer)               (None, 1, 100, 100)  0                                            
__________________________________________________________________________________________________
multiply_7 (Multiply)           (None, 1, 100, 100)  0           output[0][0]                     
                                                                 mask[0][0]                       
==================================================================================================
Total params: 28,881
Trainable params: 28,881
Non-trainable params: 0
__________________________________________________________________________________________________

Answer 2

Making this work with tensorflow 2.0 and tf.keras.

import tensorflow as tf
from tensorflow.keras.layers import Multiply, Conv2D, ConvLSTM2D, Input

main_input = Input(shape=(None, 2, 100, 100), dtype='float32', name='input')

mask=Input(shape=(1, 100, 100), dtype='float32', name='mask')

hidden = ConvLSTM2D(filters=16, 
                    kernel_size=(5, 5),  
                    padding='same', 
                    return_sequences=False, 
                    data_format='channels_first')(main_input)

output = Conv2D(filters=1, 
                kernel_size=(1, 1), 
                padding='same',
                activation='sigmoid',
                kernel_initializer='glorot_uniform',
                data_format='channels_first',
                name='output')(hidden)
output_with_mask=Multiply()([output, mask])

Answer 3

Creating an new output and use your old output as second hiden layer.

You want to make an second convolution (with an spécial mask) on your "old output" to get your new output

Hope it will help you