How do I improve the model when training has high accuracy but model.evaluate() gives low accuracy?

Question

I am working on a project that deals with sensor data. I collected the sensor data from 3 places and stored the data as a csv file. There are about 4000 samples. The data has X, Y, Z which I am using as features and class that is my target variable. The goal is to build a classification model on the dataset. The target variable/labels are [1,2,3]. Since it is a time series data, I chose to use Bidirectional LSTM.

Data distribution:

model = keras.Sequential()
model.add(
    keras.layers.Bidirectional(
    keras.layers.LSTM(
    units = 128,
    input_shape = [X_train.shape[1], X_train.shape[2]]
    )
)
)
model.add(keras.layers.Dropout(rate = 0.2))
model.add(keras.layers.Dense(units = 128, activation = 'relu'))
model.add(keras.layers.Dense(y_train.shape[1], activation = 'softmax'))
model.compile(
loss = 'categorical_crossentropy',
optimizer = 'adam',
metrics = ['acc']
)

After I train my model

history3 = model.fit(
    X_train, y_train,
    epochs=35,
    batch_size=100,
    validation_split = 0.1,
    shuffle=False
)

Here's the accuracy and validation accuracy given by my last epoch:

Epoch 35/35
4002/4002 [==============================] - 3s 858us/step - loss: 0.0216 - acc: 0.9948 - val_loss: 0.3026 - val_acc: 0.9056

When I use model.evaluate(X_test, y_test) it returns a list of two values: [5.144028138408701, 0.43551796674728394]

So the question is what are those two values? My guess is that the first value is MSE and the second is an accuracy. If I am right so why is the accuracy so low when I use .evaluate ? What should I do to improve the model??

PS More information

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

(4447, 24, 3) (4447, 3) (473, 24, 3) (473, 3)

The data is ordered data so I use shuffle = True during split.

df_train, df_test = train_test_split(df, test_size = 0.1, shuffle = True)

Answer 1

Your guess of what model.evaluate returns is correct, the first one being loss and the second one being accuracy. Seems like your model is overfitting, even though val_accuracy is showing something else. I would suggest checking how many labels of both categories there are in your dataset. What I mean by that is, maybe you have 90% of one class, while only having 10% of the other. If that is the case, there is a simple thing that you can do with sklearn(I'm guessing you used train_test_split from sklearn.model_selection).That is, you should add parameter stratify=y into train_test_split function. That can help with imbalanced datasets. Also, did you try playing with other optimization algorithms? Did you lower the learning_rate?

Answer 2

I suspect you have a highly imbalanced data set. For example if you had a data set of 10,000 samples for class 0 and 1000 samples for class 1. Assume you split that into a training set and a validation set. Both sets will still have this imbalance. Now if you train your network it will favor selecting class 0 and the validation data results will look good. If the network always predicts class 0 it will have 90% accuracy. Now if you have a test set that is balanced say 500 samples for class 0 and 500 samples for class 1 when you run model evaluate on the test set you will have a high error rate. I have not used it but I know model.fit has a parameter called class_weight.This is a dictionary that allows you adjust the impact on the loss function based on the class. In the above example you would want Class 1 samples to have 10 times the weight of the class 0 samples. You then create a class_weight dictionary as

class_weight={0:.55, 1:5.5}

Then use this in model.fit. The other things you can do are use an adjustable learning rate and save the model with the lowest validation loss and use it for evaluations. This is easy to do using Keras callbacks. The callback ModelCheckpoint can be set up to monitor the validation loss and save the model with the lowest loss. Documentation is here. The ReduceLROnPlateau callback can be set up to monitor validation loss and reduce the learning by a factor if the loss fails to decrease after N consecutive epochs. Documentation is here.