Sequence Prediction using keras LSTM

Question

I want to use LSTM for sequence prediction. I am using keras for that.

Instead of predicting next word in a sentence, I want to predict label of a line. In dataset, I have set of features and label associated with that line and bunch of documents which has set of lines.

Ex. Documents : 200 Lines in each document(VAriable but max number of lines I do zero padding at the end to match the length): 400 Features associated with each line : 100

So my input feature size is : 200 * 400 * 100

Now each line has label associated with it which i want to predict. Possible values of label = 3

So my label size : 200*400*3

I am using following code for modeling the LSTM.

model = Sequential()
model.add(LSTM(50, input_shape=(400, 100)))
model.add(Dense(3))
model.add(Activation('softmax'))
optimizer = RMSprop(lr=0.01)
model.compile(loss='categorical_crossentropy', optimizer=optimizer)

I am getting some dimensional error which says:

Error when checking model target: expected activation_3 to have 2 dimensions, but got array with shape (200L, 400L, 3L)

Answer 1

If you want to predict a label for each line instead of one label for all lines you need to pass return_sequences=True to the LSTM layer. And then you need to wrap the following non-recurrent layers with the TimeDistributed wrapper so that they can correctly deal with the returned sequence data:

model = Sequential()
model.add(LSTM(50, input_shape=(400, 100), return_sequences=True))
model.add(TimeDistributed(Dense(3)))
model.add(TimeDistributed(Activation('softmax')))