如何在 RNN 模型中使用图像集

Question

Hello I am trying to do my first RNN using Keras and Tensorflow, but I am getting stuck on an issue or reshaping my images to fit into the model.

I have looked at this post but could not figure out about the reshaping:

Keras - Input a 3 channel image into LSTM

What I have is a bunch of images that are taken at every frame in a video. I saved all the frames outside of python so I have a very large folder of images.I separated the frames into 21 frames for a segment so 21 images per motion that I want to capture. I want to read in these 21 images as one sequence. I have the same sequence captured from multiple cameras/angles which I want to us in this model. What I want to try is to model a movement and see if a person is doing this movement or not, so it is a binary model yes or no basically. Not the most sophisticated but its a learning process to use this model and keras.

I need help figuring out how to use these images inside the keras model. I have looked at a few tutorials on MINST data set but that didnt help me figure this out. Any help will be appreciated.

This is the error that is given to me when I try to train the model

ValueError: Error when checking input: expected lstm_1_input to have 3 dimensions, but got array with shape (2026, 200, 200, 1)

My code is this:

from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.layers import LSTM
from tqdm import tqdm
import cv2
import os
import numpy as np

imageSize = 200

#create lables for each image
def labelImage(img):
    wordLabel = img.split('.')[-3]
    #Conversion to one hot array [lat,not]
    if wordLabel == "FWAC":
        return[1,0]
    else:
        return[0,1]

#Process images and add lables
#Convert data into an array and add its lable
def makeTrainingData():
    print("Creating Training Data")
    trainingData = []
    for img in tqdm(os.listdir(trainDir)):
        label = labelImage(img)
        path = os.path.join(trainDir,img)
        img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
        img = cv2.resize(img, (imageSize,imageSize))
        trainingData.append([np.array(img),np.array(label)])

    #Save the array file to load it into other models if needed
    np.save("trainingData.npy", trainingData)
    print("Training Data Saved")
    return trainingData

#process the testing data in the same manner
def processTestData():
    print("Creating Testing Data")
    testData = []
    for img in tqdm(os.listdir(testDri)):
        print("image", img)
        path = os.path.join(testDri, img)
        imgNum = img.split(".")[0]
        img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
        img = cv2.resize(img, (imageSize, imageSize))
        testData.append([np.array(img), imgNum])

    np.save("testingData.npy", testData)
    print("Testing Data Saved")
    return testData



rnnSize = 512

model = Sequential()
model.add(LSTM(rnnSize, input_shape=(imageSize, imageSize)))
model.add(Dense(1024))
model.add(Activation('relu'))
model.add(Dense(50))
model.add(Activation('sigmoid'))
model.add(Dense(3))  
model.add(Activation('softmax'))


model.compile(loss='mean_squared_error', optimizer='adam',metrics=['accuracy'])

#Data
trainDir = "D:/TrainingDataSets/TrainingSet/"
testDri = "D:/TrainingDataSets/TestingSet/"

#trainData = makeTrainingData()
#testData = processTestData()
trainData = np.load('trainingData.npy')
testData = np.load("testingData.npy")
#resize the image to this See above
train = trainData[:-500]
test = trainData[-200:]

x = []
y = []
for xi in trainData:
    x.append(xi[0].reshape((-1, imageSize, imageSize)))
    y.append(xi[1])

x_train = np.array([i[0] for i in train]).reshape(-1,imageSize, imageSize,1)
y_train = [i[1] for i in train]



test_x = np.array([i[0] for i in test]).reshape(-1,imageSize , imageSize,1)
test_y = [i[1] for i in test]


epoch = 5
batchSize = 100

model.fit(x_train, y_train, epochs=epoch, batch_size= batchSize, verbose=1, shuffle=False)

Answer 1

For the error before dense layers add this line:

model.add(Flatten())

Previously, you should import:

from keras.layers import Flatten