训练精度很高，训练过程中损失少，但分类不好

Question

I train a neural network on image descriptors belonging to 3 classes (two species of animals, and one group of landscape images). These descriptors have been pre-computed with VGG16 (without the last fully-connected layers), and have given good results with other classifiers (SVM).

This is my model:

model = keras.models.Sequential()
model.add(keras.layers.Dense(256, input_shape = (25088,), activation = 'relu'))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(len(classes), activation = 'softmax'))
model.compile(optimizer = 'rmsprop', loss = 'categorical_crossentropy', metrics = ['accuracy'])

I train it like that:

model.fit(
    X,
    y,
    epochs = 50,
    batch_size = 32,
    validation_split = 0.3,
    class_weight = class_weights
)

The datasets for the three classes are imbalanced: class 0 has 2135 items, class 1 has 1472, and class 2 has 760. I use class_weights to compensate:

class_weights = {c: len(y) / np.sum(y[:,c] == 1.) for c in range(y.shape[1])}

Its value is {0: 2.045433255269321, 1: 2.9667119565217392, 2: 5.746052631578947} .

The testing accuracy and loss during training are very good (not so much on the validation sets):

Epoch 1/50
3056/3056 [==============================] - 16s 5ms/step - loss: 3.1452 - acc: 0.9107 - val_loss: 54.5996 - val_acc: 0.3997
Epoch 2/50
3056/3056 [==============================] - 2s 523us/step - loss: 1.5053 - acc: 0.9627 - val_loss: 53.9704 - val_acc: 0.4134
Epoch 3/50
3056/3056 [==============================] - 2s 521us/step - loss: 1.3939 - acc: 0.9607 - val_loss: 54.4188 - val_acc: 0.4043
Epoch 4/50
3056/3056 [==============================] - 2s 522us/step - loss: 1.5265 - acc: 0.9545 - val_loss: 53.7266 - val_acc: 0.4195
Epoch 5/50
3056/3056 [==============================] - 2s 522us/step - loss: 1.4650 - acc: 0.9562 - val_loss: 54.0863 - val_acc: 0.4111
Epoch 6/50
3056/3056 [==============================] - 2s 521us/step - loss: 1.3557 - acc: 0.9607 - val_loss: 53.8348 - val_acc: 0.4172
Epoch 7/50
3056/3056 [==============================] - 2s 520us/step - loss: 1.0602 - acc: 0.9699 - val_loss: 54.1266 - val_acc: 0.4104
Epoch 8/50
3056/3056 [==============================] - 2s 526us/step - loss: 0.8097 - acc: 0.9781 - val_loss: 55.3352 - val_acc: 0.3852
Epoch 9/50
3056/3056 [==============================] - 2s 521us/step - loss: 0.8912 - acc: 0.9741 - val_loss: 53.8360 - val_acc: 0.4172
Epoch 10/50
3056/3056 [==============================] - 2s 517us/step - loss: 0.9512 - acc: 0.9732 - val_loss: 54.1430 - val_acc: 0.4096
Epoch 11/50
3056/3056 [==============================] - 2s 519us/step - loss: 0.9200 - acc: 0.9745 - val_loss: 54.4828 - val_acc: 0.4027
Epoch 12/50
3056/3056 [==============================] - 2s 526us/step - loss: 0.7612 - acc: 0.9797 - val_loss: 53.9240 - val_acc: 0.4150
Epoch 13/50
3056/3056 [==============================] - 2s 522us/step - loss: 0.6478 - acc: 0.9820 - val_loss: 53.9454 - val_acc: 0.4150
Epoch 14/50
3056/3056 [==============================] - 2s 525us/step - loss: 0.9011 - acc: 0.9764 - val_loss: 54.3105 - val_acc: 0.4073
Epoch 15/50
3056/3056 [==============================] - 2s 517us/step - loss: 0.8652 - acc: 0.9787 - val_loss: 54.0913 - val_acc: 0.4119
Epoch 16/50
3056/3056 [==============================] - 2s 522us/step - loss: 0.7115 - acc: 0.9800 - val_loss: 54.0184 - val_acc: 0.4134
Epoch 17/50
3056/3056 [==============================] - 2s 518us/step - loss: 0.6954 - acc: 0.9804 - val_loss: 53.8322 - val_acc: 0.4172
Epoch 18/50
3056/3056 [==============================] - 2s 524us/step - loss: 0.7845 - acc: 0.9794 - val_loss: 55.1453 - val_acc: 0.3883
Epoch 19/50
3056/3056 [==============================] - 2s 520us/step - loss: 0.8089 - acc: 0.9777 - val_loss: 54.0184 - val_acc: 0.4134
Epoch 20/50
3056/3056 [==============================] - 2s 524us/step - loss: 0.6779 - acc: 0.9820 - val_loss: 54.0726 - val_acc: 0.4119
Epoch 21/50
3056/3056 [==============================] - 2s 517us/step - loss: 0.5939 - acc: 0.9840 - val_loss: 54.3102 - val_acc: 0.4073
Epoch 22/50
3056/3056 [==============================] - 2s 518us/step - loss: 0.6781 - acc: 0.9810 - val_loss: 54.1643 - val_acc: 0.4104
Epoch 23/50
3056/3056 [==============================] - 2s 514us/step - loss: 0.6912 - acc: 0.9804 - val_loss: 53.9454 - val_acc: 0.4150
Epoch 24/50
3056/3056 [==============================] - 2s 521us/step - loss: 0.6296 - acc: 0.9830 - val_loss: 54.0184 - val_acc: 0.4134
Epoch 25/50
3056/3056 [==============================] - 2s 521us/step - loss: 0.8910 - acc: 0.9748 - val_loss: 55.4755 - val_acc: 0.3814
Epoch 26/50
3056/3056 [==============================] - 2s 522us/step - loss: 0.7642 - acc: 0.9794 - val_loss: 54.3102 - val_acc: 0.4073
Epoch 27/50
3056/3056 [==============================] - 2s 519us/step - loss: 0.6787 - acc: 0.9827 - val_loss: 54.3102 - val_acc: 0.4073
Epoch 28/50
3056/3056 [==============================] - 2s 521us/step - loss: 0.6762 - acc: 0.9804 - val_loss: 53.9819 - val_acc: 0.4142
Epoch 29/50
3056/3056 [==============================] - 2s 519us/step - loss: 0.6418 - acc: 0.9823 - val_loss: 54.1996 - val_acc: 0.4096
Epoch 30/50
3056/3056 [==============================] - 2s 524us/step - loss: 0.6038 - acc: 0.9833 - val_loss: 55.0238 - val_acc: 0.3921
Epoch 31/50
3056/3056 [==============================] - 2s 524us/step - loss: 0.6223 - acc: 0.9836 - val_loss: 53.8964 - val_acc: 0.4150
Epoch 32/50
3056/3056 [==============================] - 2s 523us/step - loss: 0.6354 - acc: 0.9830 - val_loss: 54.3212 - val_acc: 0.4058
Epoch 33/50
3056/3056 [==============================] - 2s 561us/step - loss: 0.6124 - acc: 0.9840 - val_loss: 54.4909 - val_acc: 0.4035
Epoch 34/50
3056/3056 [==============================] - 2s 539us/step - loss: 0.5937 - acc: 0.9846 - val_loss: 53.9819 - val_acc: 0.4142
Epoch 35/50
3056/3056 [==============================] - 2s 524us/step - loss: 0.4993 - acc: 0.9849 - val_loss: 53.9906 - val_acc: 0.4134
Epoch 36/50
3056/3056 [==============================] - 2s 525us/step - loss: 0.5461 - acc: 0.9846 - val_loss: 53.8360 - val_acc: 0.4172
Epoch 37/50
3056/3056 [==============================] - 2s 530us/step - loss: 0.4849 - acc: 0.9859 - val_loss: 54.0580 - val_acc: 0.4119
Epoch 38/50
3056/3056 [==============================] - 2s 527us/step - loss: 0.4078 - acc: 0.9882 - val_loss: 53.9454 - val_acc: 0.4150
Epoch 39/50
3056/3056 [==============================] - 2s 526us/step - loss: 0.5824 - acc: 0.9840 - val_loss: 54.4196 - val_acc: 0.4050
Epoch 40/50
3056/3056 [==============================] - 2s 525us/step - loss: 0.4924 - acc: 0.9863 - val_loss: 54.3267 - val_acc: 0.4058
Epoch 41/50
3056/3056 [==============================] - 2s 515us/step - loss: 0.4689 - acc: 0.9876 - val_loss: 53.8725 - val_acc: 0.4165
Epoch 42/50
3056/3056 [==============================] - 2s 516us/step - loss: 0.5954 - acc: 0.9853 - val_loss: 54.4130 - val_acc: 0.4043
Epoch 43/50
3056/3056 [==============================] - 2s 521us/step - loss: 0.5741 - acc: 0.9849 - val_loss: 53.9755 - val_acc: 0.4142
Epoch 44/50
3056/3056 [==============================] - 2s 535us/step - loss: 0.4941 - acc: 0.9856 - val_loss: 53.7995 - val_acc: 0.4180
Epoch 45/50
3056/3056 [==============================] - 2s 528us/step - loss: 0.5669 - acc: 0.9827 - val_loss: 53.8360 - val_acc: 0.4172
Epoch 46/50
3056/3056 [==============================] - 2s 528us/step - loss: 0.4975 - acc: 0.9856 - val_loss: 54.0184 - val_acc: 0.4134
Epoch 47/50
3056/3056 [==============================] - 2s 533us/step - loss: 0.5870 - acc: 0.9827 - val_loss: 53.9454 - val_acc: 0.4150
Epoch 48/50
3056/3056 [==============================] - 2s 536us/step - loss: 0.4608 - acc: 0.9863 - val_loss: 53.9089 - val_acc: 0.4157
Epoch 49/50
3056/3056 [==============================] - 2s 554us/step - loss: 0.9252 - acc: 0.9777 - val_loss: 54.1243 - val_acc: 0.4104
Epoch 50/50
3056/3056 [==============================] - 2s 576us/step - loss: 0.4731 - acc: 0.9876 - val_loss: 54.2266 - val_acc: 0.4088

But when I test this model on a set of 24 images (12 from class 0 and 12 from class 2), I get unsatisfying results. These are the probabilities the model gives for images of class 0:

[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]

...and for images of class 2:

[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1.0000000e+00 1.2065205e-22 0.0000000e+00]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]
[[1. 0. 0.]]

It seems that the model is very biased towards class 0. This makes me think that I haven't used class_weight right.

Where could this bias come from?

Answer 1

Assuming you used some of your data for validation (during training) I would say you are strongly overfitting.

Your vall_acc allways stays at around 40% which is even lower than the amount of class 1 images you should have in your validation set.

3056/3056 [==============================] - 2s 576us/step - loss: 0.4731 - acc: 0.9876 - val_loss: 54.2266 - val_acc: 0.4088

In other words, your network is memorizing your training data. This can happen, amongst other things, if you have not enough data or a too complex network.

Did you choose your validation- and test-data randomly? Because if you didn't there might be a difference between those training and test data you are not aware of.