Incompatible shapes of TimeseriesGenerator feed and Dense layer - Keras/Tensorflow

Question

I have some dimension issues when I am using the Dense layer in combination with TimeseriesGenerator .

My training data looks like this:

X = (1 000 000, 6)

y = (1 000 000, 2)

I put all of this in TimeseriesGenerator :

train_gen = TimeseriesGenerator(X, y, length=4, batch_size=32, stride=1)

and I receive:

train_gen[0][0].shape
(32, 4, 6)

train_gen[0][1].shape
(32, 2)

afterwards I created a simple model:

optimizer = keras.optimizers.RMSprop()

model = Sequential()
model.add(Dense(12, input_shape=(4,6), activation='tanh'))
model.add(Dense(40, activation='tanh'))
model.add(Dense(2, activation='tanh'))

model.compile(loss='mean_absolute_error', optimizer= optimizer, metrics=['mean_absolute_error', 'accuracy'])

and the last step - fitting data:

mw = model.fit_generator(generator=train_gen, epochs=1, verbose=1)

Now I get an error. The last layer has a dimension issue:

InvalidArgumentError: Incompatible shapes: [32,4,2] vs. [32,2] [Op:Sub] name: loss/dense_44_loss/sub/

I assume the model wants to compare the [32,4,2] shaped output of the model with the given [32,2] shaped training data.

I haven't found a solution yet. I think I definitely need the TimeseriesGenerator due to the size of my original dataset which has 160 billion samples and I don't have enough RAM. Can someone help me?

Answer 1

Your last layer has a dimensionality error, which you can simply fix by adding a Flatten() layer like this:

import numpy as np
from keras.preprocessing.sequence import TimeseriesGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.optimizers import RMSprop

X = np.random.rand(996, 6)
y = np.random.rand(996, 2)

t_gen = TimeseriesGenerator(X, y, length=4, batch_size=32, stride=1)

optimizer = RMSprop()

model = Sequential()
model.add(Dense(12, input_shape=(4,6), activation='tanh'))
model.add(Dense(40, activation='tanh'))
model.add(Flatten())
model.add(Dense(2, activation='tanh'))

model.compile(loss='mean_absolute_error', 
    optimizer= optimizer, 
    metrics=['mean_absolute_error', 'accuracy'])


mw = model.fit_generator(generator=t_gen, epochs=1, verbose=1)

The result will be like this:

Epoch 1/1
31/31 [==============================] - 0s 12ms/step - loss: 0.2817 - mean_absolute_error: 0.2817 - accuracy: 0.4748

Your new model is like this:

Answer 2

Usually the TimeseriesGenerator is used for creating 3D data for timeseries applications and then you use LSTM to process this 3D data.

You can still use the TimeseriesGenerator together with Dense layers with some hacks.

Change the length of the TimeseriesGenerator to be 1 .

train_gen = TimeseriesGenerator(X, y, length=1, batch_size=32, stride=1)
#shapes : train_gen[0][0] : (32,1,6)

Now your data is essentially a 2D data and that middle dimension is just useless. So just create a Lambda layer which drops this middle dimension in your model as the first layer.

import keras.backend as K
from keras.layers import Lambda


optimizer = keras.optimizers.RMSprop()

model = Sequential()
model.add(Lambda(lambda inp: K.squeeze(inp, axis=1),input_shape=(1,6)))
model.add(Dense(12, activation='tanh'))
model.add(Dense(40, activation='tanh'))
model.add(Dense(2, activation='tanh'))
model.compile(loss='mean_absolute_error', optimizer= optimizer, metrics=['mean_absolute_error', 'accuracy'])

print(model.output_shape) #(None,2)

Your model output shape must match with your labels shape.

mw = model.fit_generator(generator=train_gen, epochs=1, verbose=1) #runs