使用 vgg16 验证集的准确性低

Question

I'm doing dog breed classification for a project and I encounter a major issue I have no idea how to solve it.

The dataset is the images of dogs provided by Stanford Dogs dataset .

I do a data augmentation with keras:

from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras import optimizers
from keras.callbacks import History 
from keras.applications import vgg16

batch_size = 16

# this is the augmentation configuration I will use for training
 train_datagen = ImageDataGenerator(rotation_range=20,
                               zoom_range=0.2,
                               horizontal_flip=True,
                               fill_mode='nearest',
                               preprocessing_function=vgg16.preprocess_input)

 # This is the augmentation configuration I will use for testing/validation... just a rescale
 test_datagen = ImageDataGenerator(rescale=1./255)

 # This is the generator which will read pictures found in my training subset
 train_generator = train_datagen.flow_from_directory('../data/train/',
                                                target_size = (224, 224),
                                                batch_size = batch_size,
                                                shuffle=True,
                                                class_mode = 'categorical',
                                                seed=42)

  # This is the generator for validation data
  validation_generator = test_datagen.flow_from_directory('../data/validation/',
                                                    target_size = (224, 224),
                                                    batch_size = batch_size,
                                                    class_mode = 'categorical')

Then I'm using transfer learning with VGG16 to train my model:

from keras.applications.vgg16 import VGG16
from keras.layers import Input, Dense, Flatten, GlobalAveragePooling2D
from keras.models import Model, Sequential

pretrained_model = VGG16(weights="imagenet", include_top=False, input_shape=(224, 224, 3))

for layer in pretrained_model.layers:
    layer.trainable = False

x = pretrained_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation="relu")(x)
predictions = Dense(120, activation='softmax')(x)

model = Model(inputs = pretrained_model.input, outputs=predictions) 

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

When I trained the model it seems like it's overfitting: I got 0.99 accuracy on train and 0.23 on validation.

When looking to the images in the train and validation set there are no apparent reasons it gives me such a bad accuracy for the validation.

What I've done is checked manually what breed the model predicts for several images (more than 50) in the vaidation and test and more than 80% are corrects predictions.

So I'm thinking there is a problem on the validation accuracy but have no idea how to fix it.

Thanks for your help !

Answer 1

There are few things that may improve the classification accuracy on training as well as validation dataset:

1. Firstly, set layer.trainable=True, because the Imagenet dataset is trained on different dataset. Just fine-tuning the top layers will directly lead to overfitting. Load imagenet weights but re-train all or couple of layers.

2. Use EfficientNet with noisy_student weights. There are less number of parameters to train. It gives better accuracy due to the scalable architecture it has. Convert the whole dataset to numpy array. I guess np array will load faster. Split the training data using sklearn train_test_split function. Normalize the dataset (1/255). Calculate np.mean along axis 0 on train data and subtract it from both train and val dataset.

3. You can use test time augmentation. In your test data generator, do a simple horizontal flip, vertical flip (if data looks realistic) and affine transformations. It will generate multiple views of the data and helps the model to average out more probable class.

4. Checkout imgaug library (embossing, sharpening, noise addition, etc.). Plus, there are random_eraser, cut out and mix up strategies that have been proved to be useful. Add this to preprocessing function rather putting preprocess_input. It will also help to regularize your model.

5. Try label smoothing. It can also help your classifier to give more probability to the correct class.

6. Try learning rate warmup. Something like this:

LR_START = 0.0001
LR_MAX = 0.00005
LR_MIN = 0.0001
LR_RAMPUP_EPOCHS = 4
LR_SUSTAIN_EPOCHS = 6
LR_EXP_DECAY = .8


def lrfn(epoch):
    if epoch < LR_RAMPUP_EPOCHS:
        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START
    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:
        lr = LR_MAX
    else:
        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN
    return lr

6. You can also extract features and apply ensemble feature classification(XGBoost, Adaboost, BaggingClassifier) or triplet loss.

7. Try increasing your image resolution to 299. Better the resolution better the fine grained classification.