过拟合 LSTM pytorch

Question

I was following the tutorial on CoderzColumn to implement a LSTM for text classification using pytorch. I tried to apply the implementation on the bbc-news Dataset from Kaggle , however, it heavily overfits, achieving a max accuracy of about 60%.

See the train/loss curve for example:

Is there any advice (I am quite new to RNN/LSTM), to adapt the model to prevent that high overfiting?

The model is taken from the above tutorial and looks kind of like this:

 class LSTMClassifier(nn.Module): def __init__(self, vocab, target_classes, embed_len = 50, hidden_dim=75, n_layers=1): super(LSTMClassifier, self).__init__() self.n_layers = n_layers self.embed_len = embed_len self.hidden_dim = hidden_dim self.embedding_layer = nn.Embedding(num_embeddings=len(vocab), embedding_dim=embed_len) # self.lstm = nn.LSTM(input_size=embed_len, hidden_size=hidden_dim,dropout=0.2, num_layers=n_layers, batch_first=True) self.lstm = nn.LSTM(input_size=embed_len, hidden_size=hidden_dim, num_layers=n_layers, batch_first=True) self.fc = nn.Linear(hidden_dim, len(target_classes)) def forward(self, X_batch): embeddings = self.embedding_layer(X_batch) hidden, carry = torch.randn(self.n_layers, len(X_batch), self.hidden_dim), torch.randn(self.n_layers, len(X_batch), self.hidden_dim) output, (hidden, carry) = self.lstm(embeddings, (hidden, carry)) return self.fc(output[:,-1])

I would be really thankful for any adive how to adapt the version in the tutorial to use it more effectively on other datasets

Answer 1

Have you tried adding nn.Dropout layer before the self.fc ? Check what p = 0.1 / 0.2 / 0.3 will do.

Another thing you can do is to add regularisation to your training via weight_decay parameter:

 optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)

Use small values first, and increase by 10 times, see which will get you the best result.

Also, goes without saying, make sure that there is no test data points in train set. Make sure you did not forget to shuffle your train set:

 train_loader = DataLoader(train_dataset, batch_size=1024, collate_fn=vectorize_batch, shuffle=True)