Combining CNN with LSTM using Tensorflow Keras

Question

I'm using pre-trained ResNet-50 model and want to feed the outputs of the penultimate layer to a LSTM Network. Here is my sample code containing only CNN (ResNet-50):

N = NUMBER_OF_CLASSES
#img_size = (224,224,3)....same as that of ImageNet    
base_model = ResNet50(include_top=False, weights='imagenet',pooling=None)
x = base_model.output
x = GlobalAveragePooling2D()(x)
predictions = Dense(1024, activation='relu')(x)
model = Model(inputs=base_model.input, outputs=predictions)

Next, I want to feed it to a LSTM network, as follows...

final_model = Sequential()
final_model.add((model))
final_model.add(LSTM(64, return_sequences=True, stateful=True))
final_model.add(Dense(N, activation='softmax'))

But I'm confused how to reshape the output to the LSTM input. My original input is (224*224*3) to CNN. Also, should I use TimeDistributed?

Any kind of help is appreciated.

Answer 1

Adding an LSTM after a CNN does not make a lot of sense, as LSTM is mostly used for temporal/sequence information, whereas your data seems to be only spatial, however if you still like to use it just use

x = Reshape((1024,1))(x)

This would convert it to a sequence of 1024 samples, with 1 feature

If you are talking of spatio-temporal data, Use Timedistributed on the Resnet Layer and then you can use convlstm2d

Answer 2

Example of using pretrained network with LSTM:

inputs = Input(shape=(config.N_FRAMES_IN_SEQUENCE, config.IMAGE_H, config.IMAGE_W, config.N_CHANNELS))
cnn = VGG16(include_top=False, weights='imagenet', input_shape=(config.IMAGE_H, config.IMAGE_W, config.N_CHANNELS))
x = TimeDistributed(cnn)(inputs)
x = TimeDistributed(Flatten())(x)
x = LSTM(256)(x)