如何在Keras中添加带LSTM的Conv2D图层？

Question

I'm trying to identify the sequence of images. I've 2 images and I need to identify the 3rd one. All are color images.

I'm getting below error:

ValueError: Error when checking input: expected time_distributed_1_input to have 5 dimensions, but got array with shape (32, 128, 128, 6)

This is my layer:

batch_size = 32
height = 128
width = 128
model = Sequential()
model.add(TimeDistributed(Conv2D(32, (3, 3), activation = 'relu'), input_shape=(batch_size, height, width, 2 * 3)))
model.add(TimeDistributed(MaxPooling2D(2, 2)))
model.add(TimeDistributed(BatchNormalization()))
model.add(TimeDistributed(Conv2D(32, (3, 3), activation='relu', padding='same')))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(LSTM(256, return_sequences=True, dropout=0.5))
model.add(Conv2D(3, (3, 3), activation='relu', padding='same'))
model.compile(optimizer='adam')
model.summary()

My input images shapes are: (128, 128, 2*3) [as I'm concatenating 2 input images]

My output image shape is: (128, 128, 3)

Answer 1

You have applied the conv layer after Flatten() . This causes error because after flattening the data flowing through the Network is no more a 2D object.

I suggest you to keep the convolutional and the recurrent phases separated. First, you apply convolution to images, training the model to extract their relevant features. Later, you push these features into LSTM layers, so that you can capture also the information hidden in their sequence.

Hope this helps, otherwise let me know.

--

EDIT :

According to the error that you get, it seems that you are also not feeding the exact input shape. Keras is saying: "I need 5 dimensions, but you gave me 4". A TimeDistributed() layers needs a shape such as: (sample, time, width, length, channel) . Your input lacks time , apparently.

I suggest you to print your model.summary() before running, and check the layer called time_distributed_1_input . That's the one your compiler is upset with.