Keras：MLP 的结果很好，但双向 LSTM 的结果很差

Question

I trained two neural networks with Keras : a MLP and a Bidirectional LSTM .

My task is to predict the words order in a sentence, so for each word, the neural network has to output a real number. When a sentence with N words is processed, the N reals number in the output are ranked in order to obtain integer numbers representing words position.

I'm using same dataset and same preprocessing on the dataset. The only different thing is that in the LSTM dataset I added padding to get the sequences of the same length.

In the prediction phase, with LSTM , I exclude the predictions created from padding vectors, since I masked them in the training phase.

MLP architecture:

mlp = keras.models.Sequential()

# add input layer
mlp.add(
    keras.layers.Dense(
        units=training_dataset.shape[1],
        input_shape = (training_dataset.shape[1],),
        kernel_initializer=keras.initializers.RandomUniform(minval=-0.05, maxval=0.05, seed=None),
        activation='relu')
    )

# add hidden layer
mlp.add(
    keras.layers.Dense(
        units=training_dataset.shape[1] + 10,
        input_shape = (training_dataset.shape[1] + 10,),
        kernel_initializer=keras.initializers.RandomUniform(minval=-0.05, maxval=0.05, seed=None),
        bias_initializer='zeros',
        activation='relu')
    )

# add output layer
mlp.add(
    keras.layers.Dense(
        units=1,
        input_shape = (1, ),
        kernel_initializer=keras.initializers.RandomUniform(minval=-0.05, maxval=0.05, seed=None),
        bias_initializer='zeros',
        activation='linear')
    )

Bidirection LSTM architecture:

model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=20, return_sequences=True), input_shape=(timesteps, features)))
model.add(Dropout(0.2))
model.add(Dense(1, activation='linear'))

The task is much better solvable with an LSTM, which should capture dependencies between words well.

However, with the MLP I achieve good results, but with LSTM the results are very bad.

Since I'm a beginner, could someone understand what is wrong with my LSTM architecture? I'm going out of head.

Thanks in advance.

Answer 1

For this problem, I am actually not surprised that MLP performs better.

The architecture of LSTM, bi-directional or not, assumes that location is very important to the structure. Words next to each other are more likely to be related than words farther away.

But for your problem you have removed the locality and are trying to restore it. For that problem, an MLP which has global information can do a better job at the sorting.

That said, I think there is still something to be done to improve the LSTM model.

---

One thing you can do is ensure that the complexity of each model is similar. You can do this easily with count_params .

mlp.count_params()
model.count_params()

If I had to guess, your LSTM is much smaller. There are only 20 units , which seems small for an NLP problem. I used 512 for a Product Classification problem to process character-level information (vocabulary of size 128, embedding of size 50). Word-level models trained on bigger data sets, like AWD-LSTM , get into the thousands of units .

So you probably want to increase that number. You can get an apples-to-apples comparison between the two models by increasing the number of units in the LSTM until the parameter counts are similar. But you don't have to stop there, you can keep increasing the size until you start to overfit or your training starts taking too long.