LSTM in Keras returns always the same output

Question

I have a dataset of size (5358,293,30) and I want to train a LSTM network to predict a value between 0 and 1.

My neural network is defined as follow:

model = Sequential()
model.add(LSTM(10, input_shape=(293, 30)))
model.add(Dense(1, activation="sigmoid"))
model.compile(loss="mean_squared_error", optimizer="adam")
model.fit(Xtrain, Ytrain, epochs=20, batch_size=38, shuffle=False)

The loss value for all the epochs during the train is ~0.04. When I test the neural network on the test data, I get always the same output as result, ~0.80. I tried a bigger network too, but the output didn't change.

I used default parameters and I scaled the data in range [0,1].

What are the possible causes for this problem? And how can I fix it?

UPDATE: The ouput of the model.summary() for the simplified version:

Layer (type)                 Output Shape              Param #   
=================================================================
lstm_1 (LSTM)                (None, 10)                1640      
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 11        
=================================================================
Total params: 1,651
Trainable params: 1,651
Non-trainable params: 0
_________________________________________________________________

And for the full version:

Layer (type)                 Output Shape              Param #   
=================================================================
lstm_2 (LSTM)                (None, 293, 64)           24320     
_________________________________________________________________
lstm_3 (LSTM)                (None, 64)                33024     
_________________________________________________________________
dense_2 (Dense)              (None, 64)                4160      
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 65        
=================================================================
Total params: 61,569
Trainable params: 61,569
Non-trainable params: 0
_________________________________________________________________

Answer 1

1. You can keep track such weird error, by using cross-validation.
2. Shuffling data help in better generalization, try it
3. Data Preparation can be another reason, have you consider shortening the sequence length, because you are using a quite large sequence.

Answer 2

If we assume your model is OK, than the first thing you may try is to increase the number of epochs.

epochs=20

Also play with the optimizer. For instance, you choose the Adam optimizer, make sure you test different parameters:

opt = Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, decay=0.01)

---

You may add the model.summary() for better anticipation of your model. I think providing the model summary is the very first thing to understand the system.

Since you mentioned features, it is very important to note how do you represent them. Based on the features representation you may need to modify the LSTM model.