Keras [文本多分类] - 训练和测试准确率高但预测差

Question

I have been facing a several problems when trying to predict topics based on news articles. The news articles have been cleared (no pontuation, numbers, ... ). There are 6 classes possible and I have a dataset of 13000 news articles per each class (Uniform distribution of the data set).

Pre-processing:

stop_words = set(stopwords.words('english')) 

for index, row in data.iterrows():
    print ("Index: ", index)
    txt_clean = ' '.join(re.sub("([^a-zA-Z ])", " ", data.loc[index,'txt_clean']).split()).lower()

    word_tokens = word_tokenize(txt_clean) 

    filtered_sentence = [w for w in word_tokens if not w in stop_words] 
    cleaned_text = ''

    for w in filtered_sentence:
        cleaned_text = cleaned_text + ' ' + w

    data.loc[index,'txt_clean'] = cleaned_text

I implemented a RNN using LSTM as the following:

model = Sequential()
    model.add(Embedding(50000, 100, input_length=500))
    model.add(SpatialDropout1D(0.2))
    model.add(LSTM(150, dropout=0.2, recurrent_dropout=0.2))
    model.add(Dense(6, activation='softmax'))
    model.summary()

    model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

    history = model.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, validation_split=0.1)

    accr = model.evaluate(X_test,Y_test)
    print('Test set\n  Loss: {:0.3f}\n  Accuracy: {:0.3f}'.format(accr[0],accr[1]))

Prediction:

model = load_model('model.h5')
data = data.sample(n=15000)

model.compile(optimizer = 'rmsprop', loss = 'categorical_crossentropy', metrics = ['accuracy'])
tokenizer = Tokenizer(num_words=50000)
tokenizer.fit_on_texts(data['txt_clean'].values) (Prediction data sample values and not the same as in the training))

CATEGORIES = ['A','B','C','D','E','F']
for index, row in data.iterrows():


    seq = tokenizer.texts_to_sequences([data.loc[index,'txt_clean']])
    padded = pad_sequences(seq, maxlen=500)

    pred = model.predict(padded)
    pred = pred[0]
    print (pred, pred[np.argmax(pred)]))

For example after 10 epochs and a batch_size of 500:

• Training acc: 0.831
• Training loss: 0.513
• Test acc: 0.714
• Test loss: 0.907

Also tried reducing the number of batch_size to 64:

• Training acc: 0.859
• Training loss: 0.415
• Test acc: 0.771
• Test loss: 0.679

The results using 64 batch size seems to me better, BUT when I am predicting news article (one by one) I get an accuracy of 15.97%. This accuracy of prediction is much much low comparing with the training and test.

What could be the problem?

Thanks!

Answer 1

This is a classical problem that exists in ML or DL. There can be a couple of reasons for it

1. Overfitting, try to make the model deeper or add some normalizations.
2. Train and Test dataset are dissimilar
3. Use the same preprocessing steps as used during training
4. Class imbalance ie training dataset contains more data for a particular class which test dataset lacks.
5. Try to change model architecture by using Bidirectional LSTM or GRU

Answer 2

Please try pickling your tokenizer using pickle or joblib to save your keras tokenizer use it for both training and prediction.

Here is the sample code to save keras tokenizer :-

import pickle

# saving
with open('tokenizer.pickle', 'wb') as handle:
    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)

# loading
with open('tokenizer.pickle', 'rb') as handle:
    tokenizer = pickle.load(handle)