Using TimeseriesGenerator with a multivariate dataset in Keras Tensorflow

Question

I am trying to model the output from a TimeseriesGenerator in Keras which is to be used as in input to the LSTM network, but have been facing issues. The dataset has the following structure:

where the set of features is shown in green (F1 to F6) and the target variable (T) shown in red. I have partitioned the total data set consisting of 3170 observations into into three sets :

Since, the LSTM in Keras requires a input size of three dimensions, I reshaped the dataset using the following command :

        train= train.reshape((train_df.shape[0], 1, train_df.shape[1]))
        validation= validation.reshape((validation.shape[0], 1, validation.shape[1]))
        test= test.reshape((test.shape[0], 1, test.shape[1]))

Thus, the size of the reshaped dataset is as follows:

where the three dimensions are (samples, timesteps, features). But the real issue is when the dataset is now passed to the timeseriesgenerator in keras. The generator code used is as follows:

        generator = TimeseriesGenerator(train, train_target, length=1, batch_size=10)

The TimeseriesGenerator passes the dataset to the fit_generator, which is as below:

        model.fit_generator(generator, validation_data=(validation, validation_target),
                                       epochs=100, verbose=0,
                                       shuffle=False, workers=1, use_multiprocessing=True)

and my LSTM network configuration in Keras is as follows:

                model = Sequential()
                model.add(LSTM(200, input_shape=(10, 6), return_sequences=True))
                model.add(LSTM(200, input_shape=(10, 6), return_sequences=False))
                model.add(Dense(1, kernel_initializer='uniform', activation='linear')) 

The input_shape to the first LSTM layer is (10,6) which means 10 samples/observations having 6 features. I choose an input_shape of (10,6) because the TimeseriesGenerator was supposed to generate a batch_size of 10 samples each having 6 features.

But, this causes an error as below:

ValueError: Error when checking input: expected lstm_input to have 3 dimensions, but got array with shape (10, 1, 1, 6)

The TimeseriesGenerator generates the input size of the train set as (10, 1, 1, 6) . The train dataset generated has four dimensions, but, I expected the TimeseriesGenerator to generate a batch_size of 10 samples with each sample having 6 features, ie a train dataset having an input size of (10,1,6) .

How do I get the TimeseriesGenerator to generate an input size of (10,1,6) ?

Answer 1

The mistake was... Dont reshape your train, validation and test data... Generator automatically make it in the shape of (10, 1,6)..because when you gave batch size 10 then it will take 10 batches... Of having length of 1 data and 6 features... Try it. It will work..

Just give your train, validation, test data in (m, n) generator will reshape it by itself