如何将 map RNN output 转换为 class 大小的张量？

Question

I'm building a binary text classifier using GLOVE embeddings and RNN. The output of nn.RNN is torch.Size([1, 12, 150]). I need to map this to a dimension of size 2 so that i can calculate loss against the actual class. Im feeding tensor of size 'words' and 100 for glove embeddings of size 100d. Eg a 12 word sentence would be of size [1,12,100]

How is this done?

def forward(self, input, hidden):

    embeds = self.embedding(input) # glove embedding
    embeds = embeds.unsqueeze(0)
    embeds = embeds.float()
    
    output, hidden = self.rnn(embeds, hidden)
    # output is size [1, 12, 150]

    return output, hidden

Answer 1

If I understand correctly, you're attempting to classify a series of words (12 in your example) with a binary class. There are many ways to do this. Essentially, you just need to reduce the size of your output with differentiable operations so that meaningful parameters are learned during training. In another sense, the RNN outputs a set of features (created from the originally input features) which are ideally more useful in classification of the text, so you essentially need to build a binary classifier which uses these features of size [number of words x 100].

For instance, you could simply average or sum all the features:

# done in two steps to avoid averaging across batch examples
pred = torch.average(output,dim = 2)
pred = torch.average(pred,dim = 1) 

You could use a fully-connected neural network layer or layers:

# in __init__()
...
self.fc = nn.Linear(100,1)


# in forward()
...
output = torch.average(output,dim = 1)
output = self.fc(output)
output = torch.nn.functional.sigmoid(output)

return output,hidden

Essentially, there's no wrong way to build the final classification module, as whatever structure you define, training the RNN will produce features that are discriminative with respect to the classification module. However, you'll likely find that some things work better than others so it's probably best to try a few things and see what works.