Why does my loss and accuracy keep going up and down with each epoch during CNN transfer learning?

Question

I am trying to use transfer learning (Densenet121) for a binary classification problem, but my accuracy and loss keep going up and down for each epoch. Is this do to low sample size (87 images) or is there something wrong with my code? I am new to all of this..

DataGen

image_size = (224,224)
batch_size = 128

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    img_dir,
    validation_split = 0.2,
    subset = "training",
    seed = 37,
    image_size = image_size,
    batch_size = batch_size,
)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    img_dir,
    validation_split = 0.2,
    subset = "validation",
    seed = 37,
    image_size = image_size,
    batch_size = batch_size,
)

DataAug

data_augmentation = keras.Sequential(
    [
        layers.RandomRotation(factor=0.15),
        layers.RandomTranslation(height_factor=0.1, width_factor=0.1),
        layers.RandomFlip(),
        layers.RandomContrast(factor=0.1),
    ]
)

Model

inputs = keras.Input(shape = (224,224,3))
x = data_augmentation(inputs)
x = tf.keras.applications.densenet.preprocess_input(x)

base_model = tf.keras.applications.DenseNet121(
    include_top=False,
    input_shape = (224,224,3),
    input_tensor = x,
    weights="imagenet",
)
base_model.trainable = False
x = layers.GlobalAveragePooling2D(name = 'avg_pool')(base_model.output)
prediction = layers.Dense(1,activation='sigmoid')(x)

new_model = (keras.Model(inputs=inputs, outputs = prediction))

Compile and Fit

new_model.compile(
    optimizer = keras.optimizers.Adam(1e-2),
    loss = keras.losses.BinaryCrossentropy(),
    metrics = [keras.metrics.BinaryAccuracy()],
)
epochs = 50
history = new_model.fit(
    train_ds,
    epochs = epochs,
    validation_data = val_ds,
    shuffle = False,
    batch_size = 8
)

Results

Answer 1

When using data augmentation the metrics (loss, accuracy) tend to be noiser since echo forward pass is done with a completely different sample.

You can reduce this noise by changing your data augmentation params, but it may hurt in performance.

That said, make sure that your examples are still valid after data augmentation is done:

• pick up several examples and manually apply the worst case transformation
• pick up several examples and apply random transformations

Visualizing those scenarios can help you understand if your params are ok. Remember that when applying data augmentation, labels must be preserved after the transformation.

Another point to consider is the usage of Adam as optimizer. May be you need to switch to a slower one, like SGD, or at least, reduce the learning rate. Maybe you can use keras AutoTuner to validate those.