检查输入时出错：预期density_1_input具有形状（3773），但数组的形状为（111，）

Question

I want to classify email with keras, I already have folders which contains emails so I want keras to identify a model that predict where to put the non-classiflied email based on what i have already classified.

So i read all the mails and create a dataframe of two columns with panda, one is a list of all the words in the mail and the other the folder where it belongs to.

After that i created x_train , y_train , x_test and y_test to train and evaluate my code. Which gave me good results, so I wanted to classfied the non-classified emails by doing the same method, read the mail tokenize it and then use pd.get_dummies and then transform it to a numpy array.

Because it looks like the predict call can only handle numpy list or numpy array.

And here is the issue, the matrix are different beacuse the number of words in the non-classified mail and my dataset are different, that leads to different shapes and so an error and I wanted to know how to solve.

I tried to use OneHotEncoder but i don't know if it's the way i use it or not but it failed

#lst = each row contains all the word of the folder in the list2
#lst2 = each row contains the path to a folder

data = pd.DataFrame(list(zip(lst, lst2)), columns=['text', 'folder'])

train_size = int(len(data) * .8)
train_posts = data['text'][:train_size]
train_tags = data['folder'][:train_size]

test_posts = data['text'][train_size:]
test_tags = data['folder'][train_size:]

model = Sequential()
model.add(Dense(16, input_shape=(vocab_size,)))
model.add(Activation('elu'))
model.add(Dropout(0.2))
model.add(Dense(32))
model.add(Activation('elu'))
model.add(Dropout(0.2))
model.add(Dense(16))
model.add(Activation('elu'))
model.add(Dropout(0.2))
model.add(Dense(num_labels))
model.add(Activation('sigmoid'))
model.summary()

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(x_train, y_train, batch_size=batch_size, epochs=100, verbose=1, validation_data=(x_test, y_test))
score = model.evaluate(x_test, y_test, batch_size=batch_size, verbose=1)

#read the non-classified mails

sentences = read_files("mail.eml")
sentences = ' '.join(sentences)
sentences = sentences.lower()
salut = unidecode.unidecode(sentences)
salut = text_to_word_sequence(salut)
salut = np.array(pd.get_dummies(salut).values)

pred = model.predict_classes(salut, batch_size=batch_size, verbose=1)

Results of the trainings: 3018/3018 [==============================] - 0s 64us/step - loss: 0.0215 - acc: 0.9949 - val_loss: 0.0217 - val_acc: 0.9950

ValueError: Error when checking input: expected dense_1_input to have shape (3773,) but got array with shape (111,)

I use 3773 words in total, which i split into x_train and x_test the training lenght is 80% of 3773 so 3018 and the rest (775) goes to the test

traning_time (fit)

3018/3018 [==============================] - 0s 67us/step - loss: 0.0225 - acc: 0.9950 - val_loss: 0.0221 - val_acc: 0.9950

test_time (evaluate)

755/755 [==============================] - 0s 25us/step

result of evaluate

Test score: 0.022089334732748024 Test accuracy: 0.9950132541309129

I forgot to say that the read_files call is just a function I made that read the file and return a list of all the words in the mail

I tested to complete the matrix of lenght 111 by adding as many columns (full of zero) to match the 3773 lenght, this does work but the matrix is for sure false and this is giving me very poor result while i have a high "accuracy" and "val_accuracy"

Please say any idea you have if you know how to solve it

Answer 1

I solved the problem the lengtg issue between the two matrix was caused because I did not use the same dictionary when I tokenized my non classified mail and the others mails.

So if anyone encounter this problem you need to use the same tokenizer during all the program.