training loss decrease at first several epochs but jump to a high value suddenly

Question

I am training a 3D Unet on a medical dataset. I am trying to overfit the model on small dataset which only has one instance. The loss decrease firstly, but suddenly jump up to a high value. I attached the loss curve below. I add batch normalization after each convolutional layer and l2 regularization to loss.
The loss function is weighted softmax cross entropy.
Optimizer is Adam and initial learning rate is 0.0001.
I am wondering why the loss can't stabilize. Does that mean the loss function is too smooth?

Answer 1

I change the way to initialize weight. The training process is very stable right now. The model is converging as expected.

Answer 2

There are a lot of factors that affect the loss function. As you already mentioned you changed the way you initialize the weights. The other factors could be the learning rate, regularization factor, the type of optimizer you are using. Though you have already mentioned some of them.

There are a few recommendations: 1. Try using dropout. 2. Try using different combinations of learning rate and regularization and plot the loss.

Also, I think it is a good idea to use Adam optimizer. It does improve the speed of the process.

At the end of the day, mostly the results are empirical and a lot depends on your dataset.