ResNet（2D 图像）与全连接网络（1D 输入）的串联

Question

I'm using a pre-built ResNet in Keras (TensorFlow 2) in the following way:

from tensorflow.keras.applications.resnet50 import ResNet50
base_model = ResNet50(weights=None, include_top=False, input_shape=(39,39,3))
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dropout(0.5)(x)
output_tensor = Dense(self.num_classes, activation='softmax')(x)
cnn_model = Model(inputs=base_model.input, outputs=output_tensor)
opt = Adam(lr=0.001)
cnn_model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy', tf.keras.metrics.AUC()])

The inputs to the model ( base_model.input ) are 39 x 39 x 3 images. In addition, I now also want to provide a 20 dimensional vector with additional information to the model (ie 20 x 1 ). I can do this in two different ways:

1. Appending the 20 dimensional vector after the GlobalAveragePooling2D step.
2. Creating an additional Fully-connected network for the 20-dimensional vector and concatenating the output of this fully-connected network to the above ResNet after the GlobalAveragePooling2D step. Here ideally both networks are trained at the same time but I don't know if that is possible.

Can I adjust my models for both options or does that not work?

Answer 1

Yes both options make sense and are possible with Keras. For #2 you could define another model which takes the 20D vector as the input and passes it through a fully connected layer, then concatenate that output with the output of your pooling layer. And for both options you'd have to adjust your final model inputs to include both the base_model input and your 20D vector.

Answer 2

This should do it, Comment out the Dense Layer to concatenate them after the global average pooling.

from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.layers import GlobalAveragePooling2D, Dropout, Dense
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Model
import tensorflow as tf


base_model = ResNet50(weights=None, include_top=False, input_shape=(39, 39, 3))
x1 = base_model.output
x1 = GlobalAveragePooling2D()(x1)
x1 = Dropout(0.5)(x1)

input_2 = tf.keras.layers.Input(shape=(20, 1))
x2 = tf.keras.layers.Flatten()(input_2)
# comment this if needed.
x2 = tf.keras.layers.Dense(16, activation='relu')(x2)

x = tf.keras.layers.Concatenate()([x1, x2])

output_tensor = Dense(self.num_classes, activation='softmax')(x)
cnn_model = Model(inputs=[base_model.input, input_2], outputs=output_tensor)
opt = Adam(lr=0.001)
cnn_model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy', tf.keras.metrics.AUC()])
print(cnn_model.summary())