How to improve the F1-score in CNN classification

Question

My project is about classification of brain mri scans. It is important that my model goes more to false positives then false negatives, because I'm working with medical data. It is better to false detect tumor then not. My question is how do i improve the f1-score.

The brain mri scan

what i tried: 1) I undersampled the tumor class -> this improved my f1-score for 1%

dataset: https://www.kaggle.com/mateuszbuda/lgg-mri-segmentation

Data balance

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt


from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix, accuracy_score
from sklearn.preprocessing import OneHotEncoder
from sklearn import preprocessing
from sklearn.utils import shuffle

import cv2


import matplotlib.image as mpimg
from skimage.io import imread, imshow
from skimage.color import rgb2gray
from skimage import data, color, io, filters, morphology,transform, exposure, feature, util
from scipy import ndimage

from sklearn.preprocessing import StandardScaler
import seaborn as sns




import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Model
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation
from tensorflow.keras.layers import Input, Dense, Dropout, Flatten, BatchNormalization,concatenate
from tensorflow.keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D, Conv2DTranspose
from tensorflow.keras import backend as K
from tensorflow.python.keras.layers.core import Dropout, Lambda
from tensorflow.python.keras.layers.core import * 
from tensorflow.keras.layers import *
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.preprocessing import image
from tensorflow.keras import regularizers

from tensorflow.keras.applications.vgg19 import VGG19
from tensorflow.keras.applications.vgg19 import preprocess_input, decode_predictions
from tensorflow.keras.preprocessing.text import Tokenizer

import glob


from tensorflow.keras.callbacks import *
# Neural network

# Model checkpoint
checkpoint_filepath = './Classification_brain_best.h5'

# Early stopping
mc = tf.keras.callbacks.ModelCheckpoint(filepath = checkpoint_filepath,save_weights_only=True, monitor='val_loss',mode='auto',save_best_only=True)
# Early stopping
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience = 50)

adam = tf.keras.optimizers.Adam(lr=0.001)

# Neural network parameters
#-------------------------
num_classes = 2
img_rows, img_cols = image_size, image_size
input_shape = (img_rows, img_cols,3) # 1 -> grijs waarden, 3 -> kleur
epochs = 500
batch_size = 64

#------------ layers ----------

OwnClassificationNN=Sequential()

OwnClassificationNN.add(Conv2D(64,(3,3),input_shape=input_shape))
OwnClassificationNN.add(Activation('relu'))
OwnClassificationNN.add(MaxPooling2D(pool_size=(2,2)))
#The first CNN layer followed by Relu and MaxPooling layers

OwnClassificationNN.add(BatchNormalization())
OwnClassificationNN.add(Dropout(0.1))

OwnClassificationNN.add(Conv2D(32,(3,3)))
OwnClassificationNN.add(Activation('relu'))
OwnClassificationNN.add(MaxPooling2D(pool_size=(2,2)))

OwnClassificationNN.add(BatchNormalization())

OwnClassificationNN.add(Dropout(0.2))

OwnClassificationNN.add(Conv2D(16,(3,3)))
OwnClassificationNN.add(Activation('relu'))
OwnClassificationNN.add(MaxPooling2D(pool_size=(2,2)))



#---------- Classification part ------------------------------
OwnClassificationNN.add(Flatten())
OwnClassificationNN.add(Dropout(0.5)) # ---->> to avoid overfitting 
#Flatten layer to stack the output convolutions from second convolution layer
OwnClassificationNN.add(Dense(8,activation='relu'))
#Dense layer of 64 neurons

OwnClassificationNN.add(Dense(num_classes,activation='softmax'))
#The Final layer with two outputs for two categories

OwnClassificationNN.compile(loss='categorical_crossentropy',optimizer=adam,metrics=['accuracy'])


**the Model for classification**

# Trainen van het CNN
history = OwnClassificationNN.fit(X_train, y_train,batch_size=batch_size, epochs=epochs, validation_data=(X_val,y_val) ,verbose=1,callbacks=[es,mc])

This is the metric of the prediction on test data

Answer 1

There are several approaches to solving the false negatives issues, the obvious one is to improve the accuracy of the model, which could be done with previously trained model (transfer learning / fine tuning), decreasing both false negatives and false positive. You could give it a try, but you already have a really nice model with high accuracy as the test metrics reveal.

However, in some special cases where the false negative are very sensitive like yours, it is worth forcing the model to reduce the false negatives in return of increasing the false positives significantly. This could be done really easily as the last layer of your neural net is a dense layer with a softmax activation function. This function works really good as the las layer activation function because it calculates the multinomial probability distribution.

Softmax activation function

In other words, your last layer is a two neuron layer, where each neuron gives you the probability of having or not having a tumor. So if the probability of th "having a tumor neuron" is above 0.5 it will classify that case as positive and vice versa. You can always change the threshold to force your model to classify to one the cases where the probability is above 0.3 or 0.4, forcing it to have less false negatives.