Shape error when training a model with Keras (Tensorflow)

Question

I am newbie to machine learning and finding tough to grasp training of model using Keras for TensorFlow. I am trying time series prediction using TensorFlow. I have a generator function which generates the training data and labels:

x_batch, y_batch = next(generator)

print(x_batch.shape)
print(y_batch.shape)

(256, 60, 9)
(256, 60, 3)

I build the model in the following way:

model = Sequential()
model.add(LSTM(128, input_shape=(None, num_x_signals,), return_sequences=True))
model.add(Dropout(0.2))
model.add(BatchNormalization())

model.add(LSTM(128, return_sequences=True))
model.add(Dropout(0.1))
model.add(BatchNormalization())

model.add(LSTM(128))
model.add(Dropout(0.2))
model.add(BatchNormalization())

model.add(Dense(32, activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(num_y_signals, activation='relu'))

opt = tf.keras.optimizers.Adam(lr=0.001, decay=1e-6)

# Compile model
model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy']
)

My model summary looks like below:

    model.summary()
  _________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_19 (LSTM)               (None, None, 128)         70656     
_________________________________________________________________
dropout_23 (Dropout)         (None, None, 128)         0         
_________________________________________________________________
batch_normalization_18 (Batc (None, None, 128)         512       
_________________________________________________________________
lstm_20 (LSTM)               (None, None, 128)         131584    
_________________________________________________________________
dropout_24 (Dropout)         (None, None, 128)         0         
_________________________________________________________________
batch_normalization_19 (Batc (None, None, 128)         512       
_________________________________________________________________
lstm_21 (LSTM)               (None, 128)               131584    
_________________________________________________________________
dropout_25 (Dropout)         (None, 128)               0         
_________________________________________________________________
batch_normalization_20 (Batc (None, 128)               512       
_________________________________________________________________
dense_12 (Dense)             (None, 32)                4128      
_________________________________________________________________
dropout_26 (Dropout)         (None, 32)                0         
_________________________________________________________________
dense_13 (Dense)             (None, 3)                 99        
=================================================================
Total params: 339,587
Trainable params: 338,819
Non-trainable params: 768

This is how I try to train the model:

tensorboard = TensorBoard(log_dir="logs/{}".format(NAME))

filepath = "RNN_Final-{epoch:02d}-{val_acc:.3f}"  # unique file name that will include the epoch and the validation acc for that epoch
checkpoint = ModelCheckpoint("models/{}.model".format(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')) # saves only the best ones

# Train model
history = model.fit_generator(
    generator=generator,
    epochs=EPOCHS,
    steps_per_epoch=100,      
    validation_data=validation_data,
    callbacks=[tensorboard, checkpoint],
)

# Score model
score = model.evaluate(validation_x, validation_y, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
# Save model
model.save("models/{}".format(NAME))

But I am getting the below error when I try to train my model:

 ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-67-f5263636596b> in <module>()
     10     steps_per_epoch=100,
     11     validation_data=validation_data,
---> 12     callbacks=[tensorboard, checkpoint],
     13 )
     14 

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   1777         use_multiprocessing=use_multiprocessing,
   1778         shuffle=shuffle,
-> 1779         initial_epoch=initial_epoch)
   1780 
   1781   def evaluate_generator(self,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
    134             'or `(val_x, val_y)`. Found: ' + str(validation_data))
    135       val_x, val_y, val_sample_weights = model._standardize_user_data(
--> 136           val_x, val_y, val_sample_weight)
    137       val_data = val_x + val_y + val_sample_weights
    138       if model.uses_learning_phase and not isinstance(K.learning_phase(), int):

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, batch_size, check_steps, steps_name, steps, validation_split)
    915           feed_output_shapes,
    916           check_batch_axis=False,  # Don't enforce the batch size.
--> 917           exception_prefix='target')
    918 
    919       # Generate sample-wise weight values given the `sample_weight` and

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training_utils.py in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
    180                            ': expected ' + names[i] + ' to have ' +
    181                            str(len(shape)) + ' dimensions, but got array '
--> 182                            'with shape ' + str(data_shape))
    183         if not check_batch_axis:
    184           data_shape = data_shape[1:]

ValueError: Error when checking target: expected dense_13 to have 2 dimensions, but got array with shape (1, 219, 3)

Answer 1

As @yhenon mentioned in the comments section since your model has some outputs for each timestep, you must use return_sequences=True for the last LSTM layer as well.

However, it is not clear what the task is (ie classification or regression). If it is a classification task you must use 'categorical_crossentropy' as the loss function (instead of 'sparse_categorical_crossentropy' which you are currently using) and use 'softmax' as the activation function of last layer.

On the other hand, if it is a regression task you need to use a regression loss such as 'mse' or 'mae' and set the activation function of last layer properly, depending on the output values (ie use 'linear' if the range of output values include both negative and positive numbers).