连接不同形状的 keras 层输出

Question

The keras model is like this:

input_x = Input(shape=input_shape)

x=Conv2D(...)(input_x)
...
y_pred1 = Conv2D(...)(x) # shape of (None, 80, 80, 2)
y_pred2 = Dense(...)(x) # shape of (None, 4)

y_merged = Concatenate(...)([y_pred1, y_pred2])


model = Model(input_x, y_merged)

y_pred1 and y_pred2 are the results I want the model to learn to predict.

But the loss function fcn1 for the y_pred1 branch need y_pred2 prediction results, so I have to concatenate the results of the two branches to get y_merged , so that fcn1 will have access to y_pred2 .

The problem is, I want to use the Concatenate layer to concatenate the y_pred1 (None, 4) output with the y_pred2 (None, 80, 80, 2) output, but I don't know how to do that.

How can I reshape the (None, 4) to (None, 80, 80, 1) ? For example, by filling the (None, 80, 80, 1) with the 4 elements in y_pred2 and zeros.

Is there any better solutions than using the Concatenate layer?

Answer 1

Maybe this extracted piece of code could help you:

tf.print(condi_input.shape)
# shape is TensorShape([None, 1])
condi_i_casted = tf.expand_dims(condi_input, 2)
tf.print(condi_i_casted.shape)
# shape is TensorShape([None, 1, 1])
broadcasted_val = tf.broadcast_to(condi_i_casted, shape=tf.shape(decoder_outputs))
tf.print(broadcasted_val.shape)
# shape is TensorShape([None, 23, 256])

When you want to broadcast a value, first think about what exactly you want to broadcast. In this example, condi_input has shape(None,1) and helped me as a condition for my encoder-decoder lstm network. To match all dimensionalities, of the encoder states of the lstm, first I had to use tf.expand_dims() to expand the condition value from a shape like [[1]] to [[[1]]] .

This is what you need to do first. If you have a prediction as a softmax from the dense layers, you might want to use tf.argmax() first, so you only have one value, which is way easier to broadcast. However, its also possible with 4 but keep in mind, that the dimensions need to match. You cannot broadcast shape(None,4) to shape(None,6) , but to shape(None,8) since 8 is devidable through 4.

Then you you can use tf.broadcast() to broadcast your value into the desired shape. Then you have two shapes, you can concatenate together. hope this helps you out.

Answer 2

Figured it out, the code is like this:

input_x = Input(shape=input_shape)
x=Conv2D(...)(input_x)
...
y_pred1 = Conv2D(...)(x) # shape of (None, 80, 80, 2)

y_pred2 = Dense(4)(x) # (None, 4)

# =========transform to concatenate:===========
y_pred2_matrix = Lambda(lambda x: K.expand_dims(K.expand_dims(x, -1)))(y_pred2) # (None,4, 1,1)

y_pred2_matrix = ZeroPadding2D(padding=((0,76),(0,79)))(y_pred2_matrix) # (None, 80, 80,1)

y_merged = Concatenate(axis=-1)([y_pred1, y_pred2_matrix]) # (None, 80, 80, 3)

The 4 elements of y_pred2 can be indexed as y_merged[None, :4, 0, 2]