LSTM has high accuracy while training, however, when I test it on the training set it seems to do very poorly

Question

So I am trying to create an LSTM that can predict the next time step of a double pendulum. The data that I am trying to train with is a (2001, 4) numpy array. (ie the first 5 rows will look like:

array([[ 1.04719755,  0.        ,  1.04719755,  0.        ],
       [ 1.03659984, -0.42301933,  1.04717544, -0.00178865],
       [ 1.00508218, -0.83475539,  1.04682248, -0.01551541],
       [ 0.95354768, -1.22094052,  1.04514269, -0.05838011],
       [ 0.88372305, -1.56345555,  1.04009056, -0.15443162]])

where each row is a unique representation of the state of the double pendulum.)

So I wanted to created an LSTM that could learn to predict the next state given the current one.

Here was my code for it so far (full_sol is the (2001, 4) matrix:

import numpy as np
from tensorflow import keras
import tensorflow as tf
# full_sol = np.random.rand(2001, 4)
full_sol = full_sol.reshape((full_sol.shape[0], 1, full_sol.shape[1]))

model = keras.Sequential()
model.add(keras.layers.LSTM(100, input_shape=(None, 4), return_sequences=True, dropout=0.2))
model.add(keras.layers.TimeDistributed(keras.layers.Dense(4, activation=tf.keras.layers.LeakyReLU(
    alpha=0.3))))
model.compile(loss="mean_squared_error", optimizer="adam", metrics="accuracy")
history = model.fit(full_sol[:-1,:,:], full_sol[1:,:,:], epochs=20)

Then when I train, I get the following results:

Epoch 1/20
63/63 [==============================] - 3s 4ms/step - loss: 1.7181 - accuracy: 0.4200
Epoch 2/20
63/63 [==============================] - 0s 4ms/step - loss: 1.0481 - accuracy: 0.5155
Epoch 3/20
63/63 [==============================] - 0s 5ms/step - loss: 0.7584 - accuracy: 0.5715
Epoch 4/20
63/63 [==============================] - 0s 5ms/step - loss: 0.5134 - accuracy: 0.6420
Epoch 5/20
63/63 [==============================] - 0s 5ms/step - loss: 0.3944 - accuracy: 0.7260
Epoch 6/20
63/63 [==============================] - 0s 5ms/step - loss: 0.3378 - accuracy: 0.7605
Epoch 7/20
63/63 [==============================] - 0s 5ms/step - loss: 0.3549 - accuracy: 0.7825
Epoch 8/20
63/63 [==============================] - 0s 4ms/step - loss: 0.3528 - accuracy: 0.7995
Epoch 9/20
63/63 [==============================] - 0s 5ms/step - loss: 0.3285 - accuracy: 0.8020
Epoch 10/20
63/63 [==============================] - 0s 5ms/step - loss: 0.2874 - accuracy: 0.8030
Epoch 11/20
63/63 [==============================] - 0s 4ms/step - loss: 0.3072 - accuracy: 0.8135
Epoch 12/20
63/63 [==============================] - 0s 4ms/step - loss: 0.3075 - accuracy: 0.8035
Epoch 13/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2942 - accuracy: 0.8030
Epoch 14/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2637 - accuracy: 0.8170
Epoch 15/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2675 - accuracy: 0.8150
Epoch 16/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2644 - accuracy: 0.8085
Epoch 17/20
63/63 [==============================] - 0s 5ms/step - loss: 0.2479 - accuracy: 0.8200
Epoch 18/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2475 - accuracy: 0.8215
Epoch 19/20
63/63 [==============================] - 0s 4ms/step - loss: 0.2243 - accuracy: 0.8340
Epoch 20/20
63/63 [==============================] - 0s 5ms/step - loss: 0.2430 - accuracy: 0.8240

So, quite high accuracy. But when I test it on the training set, the predictions aren't very good.

Eg when I predict the first value:

model.predict(tf.expand_dims(full_sol[0], axis = 0))

I get array([[[ 1.0172144 , -0.3535697 , 1.1287913 , -0.23707283]]],dtype=float32)

Instead of array([[ 1.03659984, -0.42301933, 1.04717544, -0.00178865]]) .

Where have I gone wrong?

Answer 1

I don't think you are doing anything wrong. What you are getting is still fairly close to the actual value. You can either change your choice of metric so it accurately represents the degree of error in your predictions, or you could try to increase the accuracy further.