有偏数据集上的 VGG16 验证精度低

Question

i am new to machine learning i have a retinal image dataset of about 35K images from different 5 labels.

    Vgg16 model i used for training is `img_height, img_width = 224,224
    conv_base = vgg16.VGG16(weights='imagenet', include_top=False, pooling='max', input_shape = (img_width, img_height, 3))
    # check model layers are they trainable or not.
    for layer in conv_base.layers:
        layer.trainable=True
        print(layer, layer.trainable)
    model = models.Sequential()
    model.add(conv_base)
    model.add(layers.Dense(nb_categories, activation='softmax'))
    model.summary()
    # the no. imgaes to load at each iteration
    batch_size = 32
    # only rescaling
    train_datagen =  ImageDataGenerator(
        rescale=1./255
    )
    test_datagen =  ImageDataGenerator(
        rescale=1./255
    )
    # these are generators for train/test data that will read pictures #found in the defined subfolders of 'data/'
    print('Total number of images for "training":')
    train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size = (img_height, img_width),
    batch_size = batch_size, 
    class_mode = "categorical")
    print('Total number of images for "validation":')
    val_generator = test_datagen.flow_from_directory(
    val_data_dir,
    target_size = (img_height, img_width),
    batch_size = batch_size,
    class_mode = "categorical",
    shuffle=False)
    print('Total number of images for "testing":')
    test_generator = test_datagen.flow_from_directory(
    test_data_dir,
    target_size = (img_height, img_width),
    batch_size = batch_size,
    class_mode = "categorical",
    shuffle=False)
    learning_rate = 5e-5
    epochs = 25
    checkpoint = ModelCheckpoint("25_classifier.h5", monitor = 'val_acc', verbose=1, save_best_only=True, save_weights_only=False, mode='auto', period=1)
    model.compile(loss="categorical_crossentropy", optimizer=tensorflow.optimizers.Adam(lr=learning_rate, clipnorm = 1., epsilon =1e-8), metrics = ['acc'])
history = model.fit_generator(train_generator, 
                              epochs=epochs, 
                              shuffle=True, 
                              validation_data=val_generator,
                              steps_per_epoch=120,
                              callbacks=[checkpoint])

` this model gives accuracy is:

Epoch 1/25
  2/120 [..............................] - ETA: 1:31 - loss: 0.5271 - acc: 0.8281WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.2479s vs `on_train_batch_end` time: 0.6596s). Check your callbacks.
120/120 [==============================] - ETA: 0s - loss: 0.6356 - acc: 0.7914
Epoch 00001: val_acc improved from -inf to 0.77794, saving model to 25_classifier.h5
120/120 [==============================] - 167s 1s/step - loss: 0.6356 - acc: 0.7914 - val_loss: 0.6813 - val_acc: 0.7779
Epoch 2/25
120/120 [==============================] - ETA: 0s - loss: 0.6415 - acc: 0.7880
Epoch 00002: val_acc improved from 0.77794 to 0.78278, saving model to 25_classifier.h5
120/120 [==============================] - 194s 2s/step - loss: 0.6415 - acc: 0.7880 - val_loss: 0.6530 - val_acc: 0.7828
Epoch 3/25
120/120 [==============================] - ETA: 0s - loss: 0.6485 - acc: 0.7888
Epoch 00003: val_acc did not improve from 0.78278
120/120 [==============================] - 196s 2s/step - loss: 0.6485 - acc: 0.7888 - val_loss: 0.6473 - val_acc: 0.7796
Epoch 4/25
120/120 [==============================] - ETA: 0s - loss: 0.5914 - acc: 0.8073
Epoch 00004: val_acc did not improve from 0.78278
120/120 [==============================] - 197s 2s/step - loss: 0.5914 - acc: 0.8073 - val_loss: 0.6690 - val_acc: 0.7822
Epoch 5/25
120/120 [==============================] - ETA: 0s - loss: 0.5895 - acc: 0.8033
Epoch 00005: val_acc improved from 0.78278 to 0.78791, saving model to 25_classifier.h5
120/120 [==============================] - 198s 2s/step - loss: 0.5895 - acc: 0.8033 - val_loss: 0.6388 - val_acc: 0.7879
Epoch 6/25
120/120 [==============================] - ETA: 0s - loss: 0.6060 - acc: 0.7968
Epoch 00006: val_acc did not improve from 0.78791
120/120 [==============================] - 200s 2s/step - loss: 0.6060 - acc: 0.7968 - val_loss: 0.6338 - val_acc: 0.7873
Epoch 7/25
120/120 [==============================] - ETA: 0s - loss: 0.6043 - acc: 0.7964
Epoch 00007: val_acc did not improve from 0.78791
120/120 [==============================] - 198s 2s/step - loss: 0.6043 - acc: 0.7964 - val_loss: 0.6574 - val_acc: 0.7839
Epoch 8/25
120/120 [==============================] - ETA: 0s - loss: 0.6202 - acc: 0.7969
Epoch 00008: val_acc did not improve from 0.78791
120/120 [==============================] - 197s 2s/step - loss: 0.6202 - acc: 0.7969 - val_loss: 0.6812 - val_acc: 0.7785
Epoch 9/25
120/120 [==============================] - ETA: 0s - loss: 0.5965 - acc: 0.7990
Epoch 00009: val_acc improved from 0.78791 to 0.79247, saving model to 25_classifier.h5
120/120 [==============================] - 194s 2s/step - loss: 0.5965 - acc: 0.7990 - val_loss: 0.6404 - val_acc: 0.7925
Epoch 10/25
120/120 [==============================] - ETA: 0s - loss: 0.5999 - acc: 0.8010
Epoch 00010: val_acc did not improve from 0.79247
120/120 [==============================] - 195s 2s/step - loss: 0.5999 - acc: 0.8010 - val_loss: 0.6558 - val_acc: 0.7836
Epoch 11/25
120/120 [==============================] - ETA: 0s - loss: 0.5878 - acc: 0.8068
Epoch 00011: val_acc did not improve from 0.79247
120/120 [==============================] - 199s 2s/step - loss: 0.5878 - acc: 0.8068 - val_loss: 0.6601 - val_acc: 0.7842
Epoch 12/25
120/120 [==============================] - ETA: 0s - loss: 0.5592 - acc: 0.8104
Epoch 00012: val_acc did not improve from 0.79247
120/120 [==============================] - 200s 2s/step - loss: 0.5592 - acc: 0.8104 - val_loss: 0.6473 - val_acc: 0.7899
Epoch 13/25
120/120 [==============================] - ETA: 0s - loss: 0.5719 - acc: 0.8052
Epoch 00013: val_acc did not improve from 0.79247
120/120 [==============================] - 200s 2s/step - loss: 0.5719 - acc: 0.8052 - val_loss: 0.6539 - val_acc: 0.7802
Epoch 14/25
120/120 [==============================] - ETA: 0s - loss: 0.5697 - acc: 0.8104
Epoch 00014: val_acc did not improve from 0.79247
120/120 [==============================] - 196s 2s/step - loss: 0.5697 - acc: 0.8104 - val_loss: 0.6640 - val_acc: 0.7719
Epoch 15/25
120/120 [==============================] - ETA: 0s - loss: 0.5615 - acc: 0.8141
Epoch 00015: val_acc did not improve from 0.79247
120/120 [==============================] - 192s 2s/step - loss: 0.5615 - acc: 0.8141 - val_loss: 0.6762 - val_acc: 0.7680
Epoch 16/25
120/120 [==============================] - ETA: 0s - loss: 0.5502 - acc: 0.8148
Epoch 00016: val_acc did not improve from 0.79247
120/120 [==============================] - 195s 2s/step - loss: 0.5502 - acc: 0.8148 - val_loss: 0.6522 - val_acc: 0.7871
Epoch 17/25
120/120 [==============================] - ETA: 0s - loss: 0.5348 - acc: 0.8302
Epoch 00017: val_acc did not improve from 0.79247
120/120 [==============================] - 203s 2s/step - loss: 0.5348 - acc: 0.8302 - val_loss: 0.6682 - val_acc: 0.7885
Epoch 18/25
120/120 [==============================] - ETA: 0s - loss: 0.5709 - acc: 0.8115
Epoch 00018: val_acc improved from 0.79247 to 0.79647, saving model to 25_classifier.h5
120/120 [==============================] - 201s 2s/step - loss: 0.5709 - acc: 0.8115 - val_loss: 0.6203 - val_acc: 0.7965
Epoch 19/25
120/120 [==============================] - ETA: 0s - loss: 0.5061 - acc: 0.8380
Epoch 00019: val_acc did not improve from 0.79647
120/120 [==============================] - 200s 2s/step - loss: 0.5061 - acc: 0.8380 - val_loss: 0.7082 - val_acc: 0.7888
Epoch 20/25
120/120 [==============================] - ETA: 0s - loss: 0.5309 - acc: 0.8260
Epoch 00020: val_acc did not improve from 0.79647
120/120 [==============================] - 201s 2s/step - loss: 0.5309 - acc: 0.8260 - val_loss: 0.6347 - val_acc: 0.7868
Epoch 21/25
120/120 [==============================] - ETA: 0s - loss: 0.5303 - acc: 0.8271
Epoch 00021: val_acc did not improve from 0.79647
120/120 [==============================] - 199s 2s/step - loss: 0.5303 - acc: 0.8271 - val_loss: 0.6654 - val_acc: 0.7876
Epoch 22/25
120/120 [==============================] - ETA: 0s - loss: 0.5481 - acc: 0.8193
Epoch 00022: val_acc did not improve from 0.79647
120/120 [==============================] - 198s 2s/step - loss: 0.5481 - acc: 0.8193 - val_loss: 0.6677 - val_acc: 0.7737
Epoch 23/25
120/120 [==============================] - ETA: 0s - loss: 0.5360 - acc: 0.8198
Epoch 00023: val_acc did not improve from 0.79647
120/120 [==============================] - 202s 2s/step - loss: 0.5360 - acc: 0.8198 - val_loss: 0.6521 - val_acc: 0.7948
Epoch 24/25
120/120 [==============================] - ETA: 0s - loss: 0.4920 - acc: 0.8383
Epoch 00024: val_acc improved from 0.79647 to 0.79704, saving model to 25_classifier.h5
120/120 [==============================] - 200s 2s/step - loss: 0.4920 - acc: 0.8383 - val_loss: 0.6370 - val_acc: 0.7970
Epoch 25/25
120/120 [==============================] - ETA: 0s - loss: 0.5045 - acc: 0.8299
Epoch 00025: val_acc did not improve from 0.79704
120/120 [==============================] - 200s 2s/step - loss: 0.5045 - acc: 0.8299 - val_loss: 0.6357 - val_acc: 0.7916

the val loss not decreasing like wise val accuracy not increases. i applied drouput layer but results are worst then i apply l1 and l2 regularization but it give anly 77.4% accuracy. i want atleast 90% to 95% accuracy. Please help me i stuck badly.

Answer 1

Imbalanced data sets are a common problem. For example in your case if your model just predicts level 0 it will be right 20000/35000 percent of the time. There are a number of ways to deal with the problem. Most obvious of course is to find more samples for the under represented classes. Unfortunately that is often not feasible. Next thing to try is 'under-sampling'. In that case since you have a reasonably large data set, remove some percentage of images from the over represented class. In this case say remove 50% of the images from the level 0 directory. Another method is to 'over=sample' the unrepresented class. In this case you try to create 'augmented' images for the under represented classes.You can use an image processing module, for example cv2 to create augments images and store them in the directories for the under represented classes. Things you can do for example is horizontally flip the images, change brightness etc. Tensorflow has a parameter in model.fit called class_weight to help deal with the imbalance. Documentation is here. What it does is to give more weight to under represented classes in calculation of the loss function. Below is the code for a function I developed to determine the class weights. Set the parameter dir to point to your training directory.

def get_weight_dict(dir):    
    most_samples=0
    class_weight={}
    class_list=os.listdir(dir) # dir is the directory with the training samples organized by class    
    for c in (class_list): # iterate through class directories, find number of samples in each class then find class with highest number of samples
        c_path=os.path.join(dir,c)
        if os.path.isdir(c_path):            
            length=len(os.listdir(c_path)) # determine number of samples in the class directory
            if length>most_samples:
                most_samples=length   
    for i,c in enumerate(class_list): #iterate through class directories, find number of samples in each and divide total_samples by length
        c_path=os.path.join(dir,c)
        if os.path.isdir(c_path):
            length=len(os.listdir(c_path)) # number of samples inclass directory
            class_weight[i]=most_samples/length   
            #print (i,most_samples, class_weight[i])   
    return class_weight

Then in model.fit set the call_weight parameter to the result of the function. Frankly I have not found this to be very effective. The other thing I notice you are using ModelCheckpoint. It saves the model with the lowest validation loss to a location on disk. However to use it you must load the saved model and THEN do predictions. I think you would be better off is you use the callback EarlyStopping because if you set its parameter restore_best_weights=True you wont have to load the model. Just set the number of epochs high enough so that this callback activates. Documentation for callbacks is here. I also recommend you consider using an additional callback ReduceLROnPlateau. This callback reduces the learning rate automatically if the value being monitored does not improve. My recommended code for these callbacks is shown below

rlronp=tf.keras.callbacks.ReduceLROnPlateau(monitor="val_loss",factor=0.5,patience=1,
                                            verbose=1)
es=tf.keras.callbacks.EarlyStopping(monitor="val_loss", patience=3, verbose=1,
                                   restore_best_weights=True)
callbacks=[rlronp, es]

One final comment. I do not like VGG models because they have on the order of 40 million trainable parameters so they have a heavy computational cost and cause longer training time. I use the MobilenetV2 model which has about 4 million trainable parameters and is about as accurate. You use it just like you used the VGG model.