Loss and Accuracy constant with time-series multiclass classification data

Question

I have variable length time-series data with multiclass classification. The head looks like:

0   DR_24526    1   -0.261916   0.377803    1.617511    0.311707    -0.055292   0   0.740317    0   4   1.810690    -0.375699   -1.025374   0   0.806782    0.529635    -0.577077
1   DR_24526    1   0.484744    -0.262327   -0.447281   -0.497518   -0.526008   0   0.740317    0   4   1.810690    -0.618167   -1.353477   0   0.806782    0.529635    -0.577077
2   DR_24526    1   0.484744    0.484492    2.415695    1.882432    -0.565707   0   0.740317    0   4   1.810690    -0.618167   -1.353477   0   0.806782    0.529635    -0.577077
3   DR_24526    2   0.058081    0.591180    -0.415251   -0.512043   0.131860    0   0.740317    0   4   1.810690    -0.618167   -1.353477   0   0.806782    0.529635    -0.577077
4   DR_24526    1   0.591409    0.484492    1.185172    2.287045    -0.350199   0   0.740317    0   4   1.810690    -0.618167   -1.353477   0   0.806782    0.529635    -0.577077

The first column is ID whose groups have different length. I have padded and truncated to make them of equal length.

sequences = list()

for name, group in tqdm(train_df.groupby(['ID'])):
    sequences.append(group.drop(columns=['ID']).values)

#Padding the sequence with the values in last row to max length
to_pad = 112
new_seq = []
for one_seq in sequences:
    len_one_seq = len(one_seq)
    last_val = one_seq[-1]
    n = to_pad - len_one_seq
   
    to_concat = np.repeat(one_seq[-1], n).reshape(17, n).transpose()
    new_one_seq = np.concatenate([one_seq, to_concat])
    new_seq.append(new_one_seq)
final_seq = np.stack(new_seq)

#truncate the sequence to length 60
# from tf.keras.preprocessing import sequence
seq_len = 16
final_seq=tf.keras.preprocessing.sequence.pad_sequences(final_seq, maxlen=seq_len, padding='post', dtype='float', truncating='post')

In another df there is a target column with 3 classes 0, 1, 2 with equal number of classes as ID

target = pd.get_dummies(train['DrivingStyle'])
target = np.asarray(target)

This is my model code

model = tf.keras.models.Sequential()
model.add(L.Bidirectional(L.LSTM(64, dropout=0.2, input_shape=(seq_len, 17), return_sequences=True)))
model.add(L.Bidirectional(L.LSTM(64, dropout=0.2)))
model.add(L.Dense(3, activation='softmax'))

# adam = tf.optimizers.Adam(lr=0.1, clipvalue=0.5)
# adam = tf.keras.optimizers.Adam(lr=0.001, clipvalue=0.8)
# sgd = tf.keras.optimizers.SGD(lr=1)
sgd = tf.keras.optimizers.SGD(lr=1e-4, decay=1e-6, momentum=0.9, nesterov=True)

model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

model.fit(
    final_seq,
    target,
    epochs=10,
    batch_size=84,
    callbacks=[
        tf.keras.callbacks.ReduceLROnPlateau(patience=5),
    ]
)

But my loss and accuracy are levelling to a constant value

Epoch 1/10
155/155 [==============================] - 2s 11ms/step - loss: 1.1425 - accuracy: 0.3136
Epoch 2/10
155/155 [==============================] - 2s 11ms/step - loss: 1.0670 - accuracy: 0.4461
Epoch 3/10
155/155 [==============================] - 2s 11ms/step - loss: 1.0505 - accuracy: 0.4810
Epoch 4/10
155/155 [==============================] - 2s 10ms/step - loss: 1.0463 - accuracy: 0.4882
Epoch 5/10
155/155 [==============================] - 2s 11ms/step - loss: 1.0451 - accuracy: 0.4889
Epoch 6/10
155/155 [==============================] - 2s 14ms/step - loss: 1.0437 - accuracy: 0.4904
Epoch 7/10
155/155 [==============================] - 2s 11ms/step - loss: 1.0438 - accuracy: 0.4905
Epoch 8/10
155/155 [==============================] - 2s 11ms/step - loss: 1.0426 - accuracy: 0.4920
Epoch 9/10
155/155 [==============================] - 2s 13ms/step - loss: 1.0433 - accuracy: 0.4911
Epoch 10/10
155/155 [==============================] - 2s 11ms/step - loss: 1.0419 - accuracy: 0.4909

I have tried other solutions in similar type of question. I have tried 3 hidden LSTM layers with 256 nodes but none of them working.

Data Shape

print(final_seq.shape)
print(target.shape)
(12994, 16, 17)
(12994, 3)

Answer 1

Updates Answer So, your data shape looks good. There are somethings, I would change, which might improve the result:

1. Lower your batch_size to 16 or 32, as a lower batch_size improves the accuracy
2. Use Adam again as optimizer, I dont know whether your custom SGD settings are influencing the model in a bad way
3. Check your data right before it goes into the model. There might be some preprocessing issue, which we cannot see here.
4. Maybe, the data you have just is not good enough for a proper classification. You could think about reducing the class-amount to test if one class is causing the issues, because its features are mixed with other classes.
5. As some of the features do not change during time, you could try to use a CNN. Maybe not all features are relevant for this classification, so it might be better to use less features