神经网络，随着时间的推移值得改变学习率和动力

Question

Is it worth to change learning rate after certain conditions are met? And how and why to do it? For example net will start with high learning rate and after squared error is low enough learning rate will drop for better precision or learning rate should increase to jump-out of local minima?. Wouldn't it cause over-fitting? And what about momentum?

Answer 1

Usually you should start with a high learning rate and a low momentum. Then you decrease the learning rate over time and increase the momentum. The idea is to allow more exploration at the beginning of the learning and force convergence at the end of the learning. Usually you should look at the training error to set up your learning schedule: if it got stuck, ie the error does not change, it is time to decrease your learning rate.