LSTM Time Series Prediction of Intermittent Oscillator in Tensorflow

Question

I am building an LSTM time series prediction model (in TF v=1.13.1, Keras v=2.2.4) that takes as input an intermittently oscillating time domain signal. The time between each oscillation is exponentially distributed ( beta=5 ), the oscillations have a normally distributed length ( mean length=2sec , variance=1sec ), and the frequency of each oscillation is also normally distributed ( mean frequency=22hz , variance=3hz ).

I have read nearly all of the great Stackoverflow posts by Daniel Möller on the subject of LSTMs in Keras/TF. I have also experimented with different numbers of timesteps for my model, stateful vs. stateless LSTMs, different loss functions ( mean absolute error vs. mean squared error ), and different network widths/depths. The model nearly always captures that something is oscillating but fails to recreate its intermittent signature & sinusoidal shape during the sequential prediction step.

Minimally Reproducible Example:

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# Import Data
data = np.loadtxt('./my_data.csv')

# Reshape data into batches of 500 timesteps - 1 dim per timestep
# For now, we do not split into testing/training sets
# Assume all data is for training

data_pre = data.reshape(-1, 500, 1)[:,:-1,:] # Shift input data backward by 1
data_post = data.reshape(-1, 500, 1)[:,1:,:] # Shift input data forward by 1

# Build LSTM Model for Training:

# Allow flexible number of timesteps per input (shape=(None,1))
inputs = tf.keras.layers.Input(shape=(None,1))

lstm_1 = tf.keras.layers.CuDNNLSTM(units=512, return_sequences=True)(inputs)
lstm_2 = tf.keras.layers.CuDNNLSTM(units=256, return_sequences=True)(lstm_1)

# Activate dense layer with linear activation func for regression
outputs = tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(units=1, activation='linear'))(lstm_2)

lstm_model = tf.keras.Model(inputs=inputs, outputs=outputs)
lstm_model.compile('adam', loss='mae', metrics=['mae','mse'])
lstm_model.fit(x=data_pre, y = data_post, epochs=100, batch_size=16, shuffle=False) # I have trained up to 500 epochs and while the loss decreases there is no increase in prediction performance.

# Build Stateful LSTM Model for Sample-by-Sample Prediction

# Assume 1 timestep per input of dim=1
inputs = tf.keras.layers.Inputs(shape=(1,1,1))
lstm_1 = tf.keras.layers.CuDNNLSTM(units=512, return_sequences=True, stateful=True)(inputs)
lstm_2 = tf.keras.layers.CuDNNLSTM(units=256, return_sequences=True, stateful=True)(lstm_1)

outputs = tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(units=1, activation='linear'))

prediction_model = tf.keras.Model(inputs=inputs, outputs=outputs)

# Copy weights from trained, non-stateful model:
prediction_model.set_weights(lstm_model.get_weights())

#Reset network state

prediction_model.reset_states()

#Initialize model internal state with a single sample from the input data shifted by 1 unit backwards

seed = prediction_model.predict(data_pre[0][0][None, None, :])

# Predict 20secs of data

output_array = np.zeros((10000,1,1)) # Allocate Memory

for i in range(0,10000):
    temp = prediction_model.predict(seed) # Iteratively predict next sample value
    output_array[i] = temp
    seed = temp

Plots of Raw Data:

Model Output

Answer 1

Have you thought about feeding multiple inputs through each layer? For example:
Assume you have variable x that you're feeding through your model. Reshaping the data would look something like this.

import numpy as np
look_back = 5 ## This is the number of points to include in each iteration
x = np.arange(0,100)
new_array = []
for num in range(0, len(x)-look_back):
   new_array.append(x[num:num+look_back])
new_array = np.array(new_array)

print (np.array(x).shape) ## old array = (100,)
print (new_array.shape) ## new array = (95,5)

This might help your model learn what's coming next if it has enough historical context. Based on the oscillation variance I'd say this is throwing off your network and it's simply taking the average across the board.