用于情感分析的 CNN-LSTM model 验证准确率低

Question

I am working on a project to implement CNN-LSTM sentiment analysis. Below is the code

from keras.models import Sequential
from keras import regularizers
from keras import backend as K
from keras.callbacks import ModelCheckpoint
from keras.layers import Dense, Conv1D , MaxPool1D , Flatten , Dropout 
from keras.layers import BatchNormalization
from keras import regularizers
model7 = Sequential()
model7.add(Embedding(max_words, 40,input_length=max_len)) #The embedding layer
model7.add(Conv1D(20, 5, activation='relu', kernel_regularizer = regularizers.l2(l = 0.0001), bias_regularizer=regularizers.l2(0.01)))
model7.add(Dropout(0.5))
model7.add(Bidirectional(LSTM(20,dropout=0.5, kernel_regularizer=regularizers.l2(0.01), recurrent_regularizer=regularizers.l2(0.01), bias_regularizer=regularizers.l2(0.01)))) 
model7.add(Dense(1,activation='sigmoid'))


model7.compile(optimizer='adam',loss='binary_crossentropy', metrics=['accuracy'])

checkpoint7 = ModelCheckpoint("best_model7.hdf5", monitor='val_accuracy', verbose=1,save_best_only=True, mode='auto', period=1,save_weights_only=False)
history = model7.fit(X_train_padded, y_train, epochs=10,validation_data=(X_test_padded, y_test),callbacks=[checkpoint7])

Even after adding regularizers and dropout, my model has very high validation loss and low accuracy. Epoch 3: val_accuracy improved from 0.54517 to 0.57010, saving model to best_model7.hdf5 2188/2188 [==============================] - 290s 132ms/step - loss: 0.4241 - accuracy: 0.8301 - val_loss: 0.9713 - val_accuracy: 0.5701

My train and test data: train: (70000, 7) test: (30000, 7)

train['sentiment'].value_counts() 1 41044 0 28956

test['sentiment'].value_counts() 1 17591 0 12409

Can anyone please let me know how to reduce overfitting.

Answer 1

Since your code works, I believe that your network is failing silently by 'not learning' a lot from the data. Here's a list of some of the things you can generally check:

• Is your textual data well transformed into numerical data? Is it well reprented using TF-IDF or bag of words or any other method that returns a numerical representation?

• I see that you imported batch normalization but you do not apply it. Batch norm actually helps and most importantly, does the job of regularizers since each input to each layer is normalized using the mini-batch the network has seen. So maybe remove your L2 regularizations in all layers and apply a simple batch norm instead which should reduce overfitting (also, use it without the drop out since some empirical studies show that they should not be combined together)

• Your embedding output is currently set to 40, that is 40 numerical elements of a text vector that may contain more than 10,000 elements. It seems a bit low. Try something more 'standard' such as 128 or 256 instead of 40.

• Lastly, you set the adam optimizer with all the default parameters. However, the learning rate can have a big impact on the way your loss function is computed. As I am sure you know, the gradient step uses this learning rate to progress in its calculation of the derivatives for each neuron. the default is learning_rate=0.001 . So try the following code and increase a bit the learning rate (for example 0.01 or even 0.1).

A simple example:

# define model
model = Sequential()
model.add(LSTM(32)) # or CNN
model.add(BatchNormalization())
model.add(Dense(1))

# define optimizer
optimizer = keras.optimizers.Adam(0.01)

# define loss function
loss = keras.losses.binary_crossentropy

# define metric to optimize
metric = [keras.metrics.Accuracy(name='accuracy')] # you can add more

# compile model
model.compile(optimizer=optimizer, loss=loss, metrics=metric)

Final thought : I see that you went for a combination of CNN and LSTM which has great merite. However, it is always recommended to try a simple MLP network to establish a baseline score that you may later try to beat. Does a simple MLP with 1 or 2 layers and not a lot of units produce a low accuracy score as well? If it performs better than maybe the problem is in the implementation or in the hyper parameters that you chose for the layers (or even theoretical).

I hope this answer helps and cheers!