在 keras\tensorflow 中，如何将 CNN 层添加到在 imagenet 上进行预训练的 ResNet50V2 的最后一层

Question

I am trying to drop the last layer and add a simple CNN instead like the following,

model = Sequential()
base_model = ResNet50V2(include_top=False, weights="imagenet", input_shape=input_shape, pooling="avg")
base_model.trainable = False
model = Sequential()
model.add(base_model)

# I want to add the following CNN
model.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model.add(MaxPooling2D((2, 2), padding='same'))
model.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model.add(MaxPooling2D((2, 2), padding='same'))
model.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model.add(MaxPooling2D((2, 2), padding='same'))
model.add(Flatten())
model.add(Dense(128, activation='relu', kernel_initializer='he_uniform')) 
model.add(Dense(1, activation='sigmoid'))

I don't what I am missing in making this connection that I get the following error,

model.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
  File "/project/6035234/npiran/classification/venv_clf/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py", line 522, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/project/6035234/npiran/classification/venv_clf/lib/python3.7/site-packages/tensorflow/python/keras/engine/sequential.py", line 228, in add
    output_tensor = layer(self.outputs[0])
  File "/project/6035234/npiran/classification/venv_clf/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 970, in __call__
    input_list)
  File "/project/6035234/npiran/classification/venv_clf/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1108, in _functional_construction_call
    inputs, input_masks, args, kwargs)
  File "/project/6035234/npiran/classification/venv_clf/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 840, in _keras_tensor_symbolic_call
    return self._infer_output_signature(inputs, args, kwargs, input_masks)
  File "/project/6035234/npiran/classification/venv_clf/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 878, in _infer_output_signature
    self._maybe_build(inputs)
  File "/project/6035234/npiran/classification/venv_clf/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2600, in _maybe_build
    self.input_spec, inputs, self.name)
  File "/project/6035234/npiran/classification/venv_clf/lib/python3.7/site-packages/tensorflow/python/keras/engine/input_spec.py", line 235, in assert_input_compatibility
    str(tuple(shape)))
ValueError: Input 0 of layer conv2d is incompatible with the layer: : expected min_ndim=4, found ndim=2. Full shape received: (None, 2048)

Answer 1

Update Version

If you want to use CNN that exists in ResNet50V2 , Instead of using base_model. trainable = False base_model. trainable = False for all layers, do like below and train some layers. Then use option_2 and pass it to tf.keras.layers.Flatten() .

for idx, layer in enumerate(base_model.layers):
    print(f'The name of {idx} layers is   {layer.name}')
# The name of 142 layers is   conv4_block6_out
# The name of 143 layers is   conv5_block1_1_conv
# The name of 144 layers is   conv5_block1_1_bn
...
# The name of 173 layers is   conv5_block3_add
# The name of 174 layers is   conv5_block3_out
# The name of 175 layers is   avg_pool

for layer in base_model.layers[:143]:
    layer.trainable = False

for layer in base_model.layers[143:]:
    layer.trainable = True

Old Version: You have two options:

1. Use tf.keras.layers.Reshape((2,2,512)) and reshape (None, 2048) -> (None, 2, 2 ,512) . (But in the ResNet50V2 , we have CNN why do you need more CNN!)
2. Pass output of ResNet50V2 to tf.keras.layers.Flatten() . You can try like below:

Network for option_1:

model.add(base_model)
model.add(tf.keras.layers.Reshape((2,2,512)))
model.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model.add(tf.keras.layers.MaxPooling2D((2, 2), padding='same'))
model.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model.add(tf.keras.layers.MaxPooling2D((2, 2), padding='same'))
model.add(tf.keras.layers.Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model.add(tf.keras.layers.MaxPooling2D((2, 2), padding='same'))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(512, activation='relu'))
model.add(tf.keras.layers.Dropout(rate=.2))    
model.add(tf.keras.layers.Dense(256, activation='relu'))
model.add(tf.keras.layers.Dropout(rate=.2))
model.add(tf.keras.layers.Dense(10, activation='softmax'))

Full code and Network for option_2:

import tensorflow as tf
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()
X_train = X_train.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0

base_model = tf.keras.applications.ResNet50(weights="imagenet", 
                                            include_top=False,
                                            pooling="avg", 
                                            input_shape=(32,32,3))

base_model.trainable = False

model = tf.keras.Sequential()
model.add(base_model)
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(512, activation='relu'))
model.add(tf.keras.layers.Dropout(rate=.2))    
model.add(tf.keras.layers.Dense(256, activation='relu'))
model.add(tf.keras.layers.Dropout(rate=.2))
model.add(tf.keras.layers.Dense(10, activation='softmax'))        
model.compile(loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              optimizer='Adam', 
              metrics=['accuracy'])
model.fit(X_train, y_train, batch_size=256, epochs=2, validation_split=.2)

Output:

Epoch 1/2
157/157 [==============================] - 10s 47ms/step - loss: 2.2472 - accuracy: 0.1683 - val_loss: 2.0121 - val_accuracy: 0.2772
Epoch 2/2
157/157 [==============================] - 6s 40ms/step - loss: 2.0074 - accuracy: 0.2566 - val_loss: 1.9161 - val_accuracy: 0.2934