How to combine output of Conv1D with output of Conv2D in keras

Question

I want to add a scalar value to the output of a Conv2D operation in the following way:

num_filters = 16
num_targets = 10
input_conv = layers.Conv2D(num_filters, (3, 3), activation='relu', padding='same')(input_img)
target_conv = layers.Conv1D(num_filters, num_targets, use_bias=False, activation='linear')(label)
# TODO: add output target_conv to input_conv along each of the filter dimensions
# this is like adding a scalar value for each of the filter dimensions

The input_conv output is of shape (None, 28, 28, 16) and the target_conv is of shape (None, 1, 16) , where 28 x 28 corresponds to the image dimensions and 16 is the number of filters. For each filter (each of the 16 dimensions), I want to add the corresponding target_conv value as a scalar. So imagine we are just looking at the output of the first filter for input_conv and target_conv , that gives us shapes (None, 28, 28, 1) and (None, 1, 1). I want to add the output of target_conv to the output of input_conv which would still output (None, 28, 28, 1) for the first filter, and this would then happen for each filter dimension to output (None, 28, 28, 16).

However, I am not sure how to implement this?

Answer 1

All you need to do is simply add one dimensionality to the target_conv output in order to have it in 4D. This enables you to make a simple sum between the two outputs

num_filters = 16

input_img = Input((28,28,1))
label = Input((10,1))
input_conv = Conv2D(num_filters, (3, 3), 
                    activation='relu', padding='same')(input_img) # (None,28,28,num_filters)
target_conv = Conv1D(num_filters, 10, 
                     use_bias=False, activation='linear')(label) # (None,1,num_filters)

target_conv = Reshape((1,1,num_filters))(target_conv) # (None,1,1,num_filters)
sum_filter = Add()([input_conv,target_conv]) # (None,28,28,num_filters)

m = Model([input_img,label], sum_filter)
m.summary()