Is there a way to improve val_acc?

Question

Context:

I am trying to train a image classifier on kaggle cell dataset of hopefully 0.95 val_acc. I have tried many model architectures and number of epochs, among several other hyperparameters arriving at a promising set that yields a 0.9 val_acc.

Things I tried:

• shuffled image label pair together, so correct label stays with image
• normalize images so each pixel is between 0 and 1
• added BatchNormalization() , Dropout() to reduce overfitting (now model is underfitting)
• permutations of hyperparameters tried

Problem:

The set of hyperparameters that gave the best val_acc still plateus at 0.9. I have tried many permutations, is there anything I am missing/doing wrong?

Model:

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 120, 160, 8)       224       
_________________________________________________________________
batch_normalization (BatchNo (None, 120, 160, 8)       32        
_________________________________________________________________
activation (Activation)      (None, 120, 160, 8)       0         
_________________________________________________________________
dropout (Dropout)            (None, 120, 160, 8)       0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 60, 80, 8)         584       
_________________________________________________________________
batch_normalization_1 (Batch (None, 60, 80, 8)         32        
_________________________________________________________________
activation_1 (Activation)    (None, 60, 80, 8)         0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 60, 80, 8)         584       
_________________________________________________________________
batch_normalization_2 (Batch (None, 60, 80, 8)         32        
_________________________________________________________________
activation_2 (Activation)    (None, 60, 80, 8)         0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 60, 80, 8)         0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 30, 40, 8)         584       
_________________________________________________________________
batch_normalization_3 (Batch (None, 30, 40, 8)         32        
_________________________________________________________________
activation_3 (Activation)    (None, 30, 40, 8)         0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 30, 40, 8)         584       
_________________________________________________________________
batch_normalization_4 (Batch (None, 30, 40, 8)         32        
_________________________________________________________________
activation_4 (Activation)    (None, 30, 40, 8)         0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 30, 40, 8)         0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 15, 20, 8)         584       
_________________________________________________________________
batch_normalization_5 (Batch (None, 15, 20, 8)         32        
_________________________________________________________________
activation_5 (Activation)    (None, 15, 20, 8)         0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 15, 20, 16)        3216      
_________________________________________________________________
batch_normalization_6 (Batch (None, 15, 20, 16)        64        
_________________________________________________________________
activation_6 (Activation)    (None, 15, 20, 16)        0         
_________________________________________________________________
dropout_3 (Dropout)          (None, 15, 20, 16)        0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 8, 10, 16)         6416      
_________________________________________________________________
batch_normalization_7 (Batch (None, 8, 10, 16)         64        
_________________________________________________________________
activation_7 (Activation)    (None, 8, 10, 16)         0         
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 8, 10, 16)         6416      
_________________________________________________________________
batch_normalization_8 (Batch (None, 8, 10, 16)         64        
_________________________________________________________________
activation_8 (Activation)    (None, 8, 10, 16)         0         
_________________________________________________________________
dropout_4 (Dropout)          (None, 8, 10, 16)         0         
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 4, 5, 16)          6416      
_________________________________________________________________
batch_normalization_9 (Batch (None, 4, 5, 16)          64        
_________________________________________________________________
activation_9 (Activation)    (None, 4, 5, 16)          0         
_________________________________________________________________
flatten (Flatten)            (None, 320)               0         
_________________________________________________________________
dense (Dense)                (None, 240)               77040     
_________________________________________________________________
batch_normalization_10 (Batc (None, 240)               960       
_________________________________________________________________
dropout_5 (Dropout)          (None, 240)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 162)               39042     
_________________________________________________________________
batch_normalization_11 (Batc (None, 162)               648       
_________________________________________________________________
dropout_6 (Dropout)          (None, 162)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 84)                13692     
_________________________________________________________________
batch_normalization_12 (Batc (None, 84)                336       
_________________________________________________________________
dropout_7 (Dropout)          (None, 84)                0         
_________________________________________________________________
dense_3 (Dense)              (None, 4)                 340       
=================================================================
Total params: 158,114
Trainable params: 156,918
Non-trainable params: 1,196

visualization of activations and val_acc, val_loss

Note:

The optimization was done using talos which can be found here . I edited and added some modules here .

---

Edit 1:

The optimizer I used is Nadam with learning rate 0.0002. Full notebook .

TLDR:

Trained on kaggle cell dataset using best hyperparameters from a test run of trying around 200 different hyperparameters. Plateaus at 0.9. Why not higher?

full notebook

Answer 1

From what I can find, I was using too low a learning rate. Increasing it seems to help.