在LSTM中提供CNN的输出

Question

It is the first time that I am working with the LSTM networks. I have a video with a frame rate of 30 fps. I have a CNN network (AlexNet based) and I want to feed the last layer of my CNN network into the recurrent network (I am using tensorflow). Supposing that my batch_size=30 , so equal to the fps, and I want to have a timestep of 1 second (so, every 30 frames). The output of the last layer of my network will be [bast_size, 1000] , so in my case [30, 1000] , now do I have to reshape the size of my output to [batch_size, time_steps, features] (in my case: [30, 30, 1000] )? Is that correct? or am I wrong?

Answer 1

Consider to build your CNN model with Conv2D and MaxPool2D layers, until you reach your Flatten layer, because the vectorized output from the Flatten layer will be you input data to the LSTM part of your structure.

So, build your CNN model like this:

model_cnn = Sequential()
model_cnn.add(Conv2D...)
model_cnn.add(MaxPooling2D...)
...
model_cnn.add(Flatten())

Now, this is an interesting point, the current version of Keras has some incompatibility with some TensorFlow structures that will not let you stack your entire layers in just one Sequential object.

So it's time to use the Keras Model Object to complete you neural network with a trick:

input_lay = Input(shape=(None, ?, ?, ?)) #dimensions of your data
time_distribute = TimeDistributed(Lambda(lambda x: model_cnn(x)))(input_lay) # keras.layers.Lambda is essential to make our trick work :)
lstm_lay = LSTM(?)(time_distribute)
output_lay = Dense(?, activation='?')(lstm_lay)

And finally, now it's time to put together our 2 separated models:

model = Model(inputs=[input_lay], outputs=[output_lay])
model.compile(...)

Now, on our OpenCV part , use an algorithm like the one shown below to preprocess your videos directly in order to build a big tensor of frames to you feed in your network:

video_folder = '/path.../'
X_data = []
y_data = []
list_of_videos = os.listdir(vide_folder)

for i in list_of_videos:
    #Video Path
    vid = str(video_folder + i) #path to each video from list1 = os.listdir(path)
    #Reading the Video
    cap = cv2.VideoCapture(vid)
    #Reading Frames
    #fps = vcap.get(5)
    #To Store Frames
    frames = []
    for j in range(40): #here we get 40 frames, for example
        ret, frame = cap.read()
        if ret == True:
            print('Class 1 - Success!')
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) #converting to gray
            frame = cv2.resize(frame,(30,30),interpolation=cv2.INTER_AREA)
            frames.append(frame)
        else:
            print('Error!')
    X_data.append(frames) #appending each tensor of 40 frames resized for 30x30
    y_data.append(1) #appending a class label to the set of 40 frames
X_data = np.array(X_data)
y_data = np.array(y_data) #ready to split! :)

Just train it! :)

Answer 2

If you merge several small sequences from different videos to form a batch, the output of the last layer of your model (the RNN) should already be [batch_size, window_size, num_classes]. Basically, you want to wrap your CNN with reshape layers which will concatenate the frames from each batch:

• input -> [batch_size, window_size, nchannels, height, width]
• reshape -> [batch_size * window_size, nchannels, height, width]
• CNN -> [batch_size * window_size, feat_size]
• reshape -> [batch_size, window_size, feats_size]
• RNN -> [batch_size, window_size, num_outputs] (assuming frame-wise predictions)

But this will take a lot of memory, so you can set batch size to 1, which is what you seem to be doing if I understood correctly. In this case you can spare the first reshape.

I'm not sure about the order of the axes above, but the general logic remains the same.

As a side note: if you plan on using Batch Normalization at some point, you may want to raise the batch size because consecutive frames from a single segment might not contain a lot of variety by themselves. Also give a double check on the batch normalization axes which should cover both time and batch axes.