Pytorch：LSTM 不学习

Question

I'm having a hard time training my LSTM model, it does not seem to learn at all. The training loss is hardly decreasing and accuracy changes for very simple models (1 layer, few lstm units) but eventually gets stuck at 45%, just like the more complex models right from the start. Most of the times, it only predicts one class as output. I've tried varying all the hyper parameters, but it doesn't really seem to change anything so I'm afraid I'm missing something obvous.

Here is my model:

Embedding_Model(nn.Module):
    def __init__(self, dim, vocab_size, classes, lstm_units=100, num_layers=3, bidirectional=True):
        super(Embedding_Model, self).__init__()

        self.word_embeddings = nn.Embedding(vocab_size, dim)
        self.lstm =  nn.LSTM(dim, lstm_units, num_layers=num_layers, bidirectional=bidirectional, batch_first=True)
        self.fc = nn.Sequential(
            nn.Linear(lstm_units*(1+int(bidirectional)), classes),
            nn.Softmax(dim=1)
        )

    def forward(self, x):
        x = self.word_embeddings(x)
        x, (h,c)= self.lstm(x)
        x = x.transpose(0,1)
        x = self.fc(x[-1])
        return x

My training set contains roughly 5000 input sequences. The length of the input sequences is 1400 (padded with zeros). There are 150,000 different tokens, I've tried embedding dimensions between 10 and 200. I use 3 output classes (quite balanced in the training set) and cross entropy loss.

Did I mess up anything obvious? I understand that the training set is rather small, but at least I would expect some overfitting. But the model does not seem to learn anything at all.

Answer 1

You are saying that you use nn.CrossEntropyLoss as the loss function, which applies log-softmax, but you are also applying nn.Softmax in the model.

nn.CrossEntropyLoss expects the raw logits, so you need to remove the nn.Softmax from your model.

self.fc = nn.Linear(lstm_units*(1+int(bidirectional)), classes)