Transfer learning InceptionV3 show poor validation accuracy with training

Question

## MODEL IMPORTING ##

import tensorflow 
import pandas as pd
import numpy as np
import os
import keras
import random
import cv2
import math
import seaborn as sns

from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split

import matplotlib.pyplot as plt

from tensorflow.keras.layers import Dense,GlobalAveragePooling2D,Convolution2D,BatchNormalization
from tensorflow.keras.layers import Flatten,MaxPooling2D,Dropout

from tensorflow.keras.applications import InceptionV3
from tensorflow.keras.applications.densenet import preprocess_input

from tensorflow.keras.preprocessing import image
from tensorflow.keras.preprocessing.image import ImageDataGenerator,img_to_array

from tensorflow.keras.models import Model

from tensorflow.keras.optimizers import Adam

from tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau

import warnings
warnings.filterwarnings("ignore")

WIDTH = 299
HEIGHT = 299

CLASSES = 4

base_model = InceptionV3(weights='imagenet', include_top=False)

for layer in base_model.layers:
     layer.trainable = False

x = base_model.output
x = GlobalAveragePooling2D(name='avg_pool')(x)
x = Dropout(0.4)(x)
predictions = Dense(CLASSES, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=predictions)

model.summary()

model.compile(optimizer='adam',  ##also tried other optimiser --> similar poor accuracy found
          loss='categorical_crossentropy',
          metrics=['accuracy'])


## IMAGE DATA GENERATOR ##

from keras.applications.inception_v3 import preprocess_input
train_datagen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
     rotation_range=40,
        width_shift_range=0.2,
       height_shift_range=0.2,
       shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest',
      validation_split=0.2)



train_generator = train_datagen.flow_from_directory(
   TRAIN_DIR,
    target_size=(HEIGHT, WIDTH),
    batch_size=BATCH_SIZE,
    class_mode='categorical',
    subset="training")

validation_generator = train_datagen.flow_from_directory(
    TRAIN_DIR,
     target_size=(HEIGHT, WIDTH),
     batch_size=BATCH_SIZE,
     class_mode='categorical',
     subset="validation")


test_datagen = ImageDataGenerator(rescale=1./255)

generator_test = test_datagen.flow_from_directory(directory=TEST_DIR,
                                              target_size=(HEIGHT, WIDTH),
                                              batch_size=BATCH_SIZE,
                                              class_mode='categorical',
                                              shuffle=False)

## MODEL training ##

EPOCHS = 20
STEPS_PER_EPOCH = 320 #train_generator.n//train_generator.batch_size
VALIDATION_STEPS = 64 #validation_generator.n//validation_generator.batch_size
history = model.fit_generator(
    train_generator,
   epochs=EPOCHS,
steps_per_epoch=STEPS_PER_EPOCH,
validation_data=validation_generator,
validation_steps=VALIDATION_STEPS)

Result found:

VALIDATION ACCURACY around 0.55-0.67 fluctuating.. TRAINING ACCURACY 0.99

Questions:

1. What/Where is the problem in transfer learning process?
2. are train, validate and test data generator function parameter are chosen correctly?

Answer 1

Well I think you would be better off to train the entire model. So remove the code that makes the base model layers not trainable. If you look at the documentation for Inceptionv3 located here you can set pooling='max' which puts a GlobalMaxPooling2d layer as the output layer so if you do that you do not need to add your own layer as you did. Now I noticed you imported the callbacks ModelCheckpoint and ReduceLROnPlateau but you did not use them in model.fit. Using an adjustable learning rate will be beneficial to achieving a lower validation loss. ModelCheckpoint is useful to save the best model for use in predictions. See code below for implementations. save_loc is the directory where you want to store the results from ModelCheckpoint. NOTE in ModelCheckpoint I set save_weights_only=True. Reason is this is far faster than saving the entire model on each epoch for which the validation loss decreases.

checkpoint=tf.keras.callbacks.ModelCheckpoint(filepath=save_loc, monitor='val_loss', verbose=1, save_best_only=True,
        save_weights_only=True, mode='auto', save_freq='epoch', options=None)
lr_adjust=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss", factor=0.5, patience=1, verbose=1, mode="auto",
        min_delta=0.00001,  cooldown=0,  min_lr=0)
callbacks=[checkpoint, lr_adjust]
history = model.fit_generator( train_generator, epochs=EPOCHS,
          steps_per_epoch=STEPS_PER_EPOCH,validation_data=validation_generator,
          validation_steps=VALIDATION_STEPS, callbacks=callbacks)
model.load_weights(save_loc) # load the saved weights
# after this use the model to evaluate or predict on the test set.
# if you are satisfied with the results you can then save the entire model with
model.save(save_loc)

Be careful on the test set generator. Ensure you do the same preprocessing on the test data as you did on the training data. I noticed you only rescaled the pixels. Not sure what the preprocess function does but I would use that. I would also remove the dropout layer initially. Monitor the training loss and validation loss on each epoch and plot the results. If the training loss continues to decrease and the validation loss trends toward increasing then you are over fitting if so then restore the dropout layer if needed. If you do evaluation or prediction on the test set, you only want to go through the test set once. So select the test batch size to that no. of test samples/test batch size is an integer and use that integer as the number of test steps. Here is a handy function that will determine that for you where length is the number of test samples and b_max is the maximum batch size you will allow based on your memory capacity.

def get_bs(length,b_max):
    batch_size=sorted([int(length/n) for n in range(1,length+1) if length % n ==0 and length/n<=b_max],reverse=True)[0]  
    return batch_size,int(length/batch_size)
# example of use
batch_size, step=get_bs(10000,70)
#result is batch_size= 50  steps=200