How to improve prediction with keras and tensorflow

Question

I am using tensorflow with keras to perform regression on some historical data. Data type as follows:

id,timestamp,ratio "santalucia","2018-07-04T16:55:59.020000",21.8 "santalucia","2018-07-04T16:50:58.043000",22.2 "santalucia","2018-07-04T16:45:56.912000",21.9 "santalucia","2018-07-04T16:40:56.572000",22.5 "santalucia","2018-07-04T16:35:56.133000",22.5 "santalucia","2018-07-04T16:30:55.767000",22.5

And I am reformulating it as a time series problem (25 time steps) so that I can predict (make a regression) for the next values of the series (variance should not be high). I am using also sklearn.preprocessing MinMaxScaler to scale the data to range (-1,1) or (0,1) depending if I use LSTM or Dense (respectively). I am training with two different architectures:

Dense is as follows:

def get_model(self, layers, activation='relu'):
    model = Sequential()
    # Input arrays of shape (*, layers[1])
    # Output = arrays of shape (*, layers[1] * 16)
    model.add(Dense(units=int(64), input_shape=(layers[1],), activation=activation))
    model.add(Dense(units=int(64), activation=activation))
    # model.add(Dropout(0.2))

    model.add(Dense(units=layers[3], activation='linear'))
    # activation=activation))

    # opt = optimizers.Adagrad(lr=self.learning_rate, epsilon=None, decay=self.decay_lr)
    opt = optimizers.rmsprop(lr=0.001)
    model.compile(optimizer=opt, loss=self.loss_fn, metrics=['mae'])
    model.summary()
    return model

Which more or less provides with good results (same architecture as in tensorflows' tutorial for predicting house prices).

However, LSTM is not giving good results, it usually ends up stuck around a value (for example, 40 (40.0123123, 40.123123,41.09090...) and I do not see why or how to improve it. Architecture is:

def get_model(self, layers, activation='tanh'):
    model = Sequential()
    # Shape = (Samples, Timesteps, Features)
    model.add(LSTM(units=128, input_shape=(layers[1], layers[2]),
                   return_sequences=True, activation=activation))

    model.add(LSTM(64, return_sequences=True, activation=activation))

    model.add(LSTM(layers[2], return_sequences=False, activation=activation))
    model.add(Dense(units=layers[3], activation='linear'))
    # activation=activation))

    opt = optimizers.Adagrad(lr=0.001, decay=self.decay_lr)
    model.compile(optimizer=opt, loss='mean_squared_error', metrics=['accuracy'])
    model.summary()
    return model

I currently train with a batch size of 200 that increases by a rate of 1.5 every fit. Each fit is made of 50 epochs, and I use a keras earlystopping callback with at least 20 epoch.

I have tried adding more layers, more units, reducing layers, units, increasing and decreasing learning rate, etc, but every time it gets stuck around a value. Any reason for this?

Also, do you know any good practices that can be applied to this problem?

Cheers

Answer 1

Have you tried holding back a validation set seeing how well the model performance on the training set tracks with the validation set? This is often how I catch myself overfitting.

A simple function for doing this ( adapted from here ) can help you do that:

hist = model.fit_generator(...)
def gen_graph(history, title):
    plt.plot(history.history['categorical_accuracy'])
    plt.plot(history.history['val_categorical_accuracy'])
    plt.title(title)
gen_graph(hist, "Accuracy, training vs. validation scores")

Also, do you have enough samples? If you're really, really sure that you have done as much as you can in terms of preprocessing, and in terms of hyperparameter tuning... generating some synthetic data or doing some data augmentation has occasionally helped me.