Questions that in case of fluctuating the validation accuracy and loss curve for image binary classification, ask the way of analysis and solution

Question

I implement training and evaluating for binary classification with image data through transfer learning from keras API. I'd like to compare performance each models(ResNet, Inception, Xception, VGG, Efficient Net). The datasets are composed by train(approx.2000ea), valid(approx.250ea), test(approx.250ea). But I faced unfamiliar situation for me so I'm asking couple of questions here.

1. As shown below, Valid Accuracy or Loss has a very high up and down deviation. I wonder which one is the problem and what needs to be changed.

   epoch_acc_loss loss_epoch acc_epoch

2. If I want to express validation accuracy with number, what should I say in the above case? Average or maximum or minimum?

3. It is being performed using Keras (tensorflow), and there are many examples in the API for train, valid but the code for Test(evaluation?) is hard to find. When figuring performance,
   normally implement until valid? or Do I need to show evaluation result?

4. Now I use Keras API for transfer learning and set this.

   include_top=False conv_base.trainable=False Summary

   I wonder if there is an effect of transfer learning without includint from top, or if it's not,
   is there a way to freeze or learn from a specific layer of conv_base.

I'm a beginner and have not many experience so it could be ridiculous questions but please give kind advice.

Thanks a lot in advance.

Answer 1

1. It's hard to figure out the problem without any given code/model structure. From your loss graph I can see that your model is facing underfitting (or it has a lots of dropout). Common mistakes, that make models underfit are: very high lr and primitive structure (so model can't figure out the dependencies in your data). And you should never forget about the principle "garbage in - garbage out", so double-check tour data for any structure roughness.

2. Well, validation accuracy in you model training logs is mean accuracy for validation set. Validation technique is based on statistics - you take random N% out of your set for validation, so average is always better if we're talking about multiple experimets (or cross validation).

3. I'm not sure if I've understood your question correct here, but if you want to evaluate your model with the metric, that you've specified for it after the training process ( fit() function call) you should use model.evaluate(val_x, val_y) . Or you may use model.predict(val_x) and compare its results to val_y via your metric function.

4. If you are using default weights for keras pretrained models (imag.net weights) and you want to use your own fully-connected part with it, you may use ONLY pretrained feature extractor (conv blocks). So you specify include_top=False . Of course there will be some positive effect (I'd say it will be significant in comparison with randomly initialized weights) because conv blocks have params that were trained to extract correct features from image. Also would recommend here to use so called "fine-tuning" technique - freeze all layers in pretrained part except a few in its end (may be few layers or even 2-3 conv blocks). Here's the example of fine-tuning of EfficientNetB0:

    ef.net = EfficientNetB0(weights="imag.net", include_top=False, input_shape=(540, 960, 3)) ef.net.trainable = True for layer in ef.net.layers: if 'block7a' not in layer.name and 'top' not in layer.name: layer.trainable = False

   Here I freeze all pretrained weights except last conv block ones. I've looked into the model with ef.net.summary() and selected names of blocks that I want to unfreeze.