如何使用OpenCV在视频中显示CNN-LSTM预测输出？

Question

everyone,

I have a CNN-LSTM model trained in keras. As input, i loaded sets of 15 frames per video with 30x30 and with just one channel (15, 30, 30, 1).

I extracted them from a total of 279 videos, and stored them in a big tensor with dimensions (279, 15, 30, 30, 1).

X_data.shape = (279, 15, 30, 30, 1)
y_data.shape = (279,)

I'm working with two classes of videos (so targets are 0 and 1).

The input layer of my time distributed CNN (before my LSTM layer) is:

input_layer = Input(shape=(None, 30, 30, 1))

Ok, they feeded in my network and everything worked well, but now i need to predict these videos and i want to display the output in the video i'm classifying.

I wrote this to read the video and display the text:

vid = cv2.VideoCapture(video_path)

while(vid.isOpened()):
    ret, frame = vid.read()
    if ret == True:
        texto = predict_video(frame)
        frame = cv2.resize(frame,(750,500),interpolation=cv2.INTER_AREA)
        frame = cv2.putText(frame,str(texto),(0,130), cv2.FONT_HERSHEY_SIMPLEX, 2.5, (255, 0, 0), 2, cv2.LINE_AA)
        cv2.imshow('Video', frame)

        if cv2.waitKey(25) & 0xFF == ord('q'):
            break
    else:
        break

vid.release()
cv2.destroyAllWindows()

The predict_video() is used to generate the predicted output as a text, as you can see:

def predict_video(frame):
    count_frames = 0
    frame_list = []

    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    frame = cv2.resize(frame,(30,30),interpolation=cv2.INTER_AREA)

    while count_frames < 15:
        frame_list.append(frame)
    count_frames = 0

    frame_set = np.array(frame_list)
    frame_set = frame_set.reshape(1, 15, 30, 30, 1)

    pred = model.predict(frame_set)
    pred_ = np.argmax(pred,axis=1) #i'm using the Model object from Keras

    if pred_ == 1:
        return 'Archery'
    elif pred_ == 0:
        return 'Basketball'

Due to the fact that the input dimension of the CNN-LSTM is equal to (None, 30, 30, 1) i need to predict with model.predict(sample) a sample with dimensions like this (1, 15, 30, 30, 1). How i can predict a video in real time, once i want to predict not frame by frame but with a model based on sets of 15 frames?

The actual predict_video() function "freeze" my computer.

Thanks for the attention!

Answer 1

Here is piece of code you can use to put text on each frame

cv2.putText(img, text, (textX, textY ), font, 1, (255, 255, 255), 2)

Here "img" is your frame, "text" is your output prediction while "textX and textY" are your coordinates around which you want to center the text. While answer to your other part where you to make prediction on set of 15 frame rather than on a single frame. well what you can do is train a model in keras by setting batch size to 15 images and with true label present for each frame.. After you complete training model will expect you input batch of 15 frames. what you can do later in while loop is set a check that when frames passed are equal to 15 you collect those frames create a tensor of dimension (15,30,30,1)

This part of code

frame_list = []
while count_frames < 15:
    frame_list.append(frame)

should not be inside function because for every frame when function is called frame_list is set to zero since its scope is limited to function. You should write this code inside for loop and when number of frames in frame_list is equal to 15 then you should call the model.predict(batch) function and also extend the dimension first to set dimensions to (1,15,30,30,1)