使用嵌入的Keras LSTM语言模型

Question

I am doing a language model using keras.

Basically, my vocabulary size N is ~30.000, I already trained a word2vec on it, so I use the embeddings, followed by LSTM, and then I predict the next word with a fully connected layer followed by softmax. My model is written as below :

EMBEDDING_DIM = 256
embedding_layer = Embedding(N,EMBEDDING_DIM,weights=[embeddings],
trainable=False)

model = Sequential()
model.add(embedding_layer)
model.add(LSTM(EMBEDDING_DIM))
model.add(Dense(N))   
model.add(Activation('softmax')) 

model.compile(loss="categorical_crossentropy", optimizer="rmsprop")

I have two questions :

1. in this case, can you confirm that we only use the last hidden layer of the LSTM (which is followed by the fully connected layer and softmax) and there isn't something like a max/mean-pooling of successive hidden layers of the lstm (like here for sentiment analysis http://deeplearning.net/tutorial/lstm.html ) ?

2. What do you think, instead of connecting the last hidden layer of the lstm to a big fully connected layer of size N (30.000), connecting to a layer of size EMBEDDING_DIM, and predicting the embedding of the next word instead of the word itself, in which case we replace the loss by something like mse, reducing training time, and mainly "helping" our model because the vocabulary is big and embeddings can be useful also for the end of the network ?

Thanks !

Answer 1

I can only reply for sure to the first question:

Yes, the output of the LSTM layer is the last hidden unit. It only returns all the hidden states if you give it the parameter return_sequences=True . It is set to False by default.

For the second question, I can only say that I have tried to predict the embeddings insetead of the one-hot vector representation of the word but it gave me bad results. Words are still categorical variable, even if we can somehow approximate them by a continuous representation. People have put a lot of effort into developping Hierachical Softmax for this reason.