Cannot improve the accuracy of my Multilayer Perceptron (MLP) model in classification

Question

I am trying to predict labels for building performance: {1, 0}. Since this is a binary classification, I tried sigmoid and identity activation functions with Xavier initialization. However, I cannot improve the accuracy of my models as the loss and accuracy stay still after training each epoch. This is a very imbalanced dataset where the ones have the 90% majority. So, I assume this might be due to the initial bias. Can you help me with this one? You can see the setup of the training process and the other relevant images attached. model definition , hyperparameters , results

Answer 1

Here are several suggestions which may help:

1. Use activation after each hidden layer
2. Learning rate of 0.1 is too high for Adam. Try smaller (3e-4 for example)
3. You are printing loss value incorrectly. Currently loss value is taken for the last iteration only. Calculate mean epoch loss instead.
4. Minor suggestion: since the task is binary classification it's better to apply torch.nn.BCELoss or torch.nn.BCEWithLogitsLoss if you don't use sigmoid on last layer. Last linear layer must have output_size=1 in this case.
5. Best model checkpoint may be missed with code you provided. That's because you calculate accuracy each 10 epochs, however accuracy > best_accuracy is done on each epoch which is inconsistent.