人工神经网络软件程序中的学习率参数

Question

I am using a software program of the type that is known as an Artificial Neural Network. One of the parameters of the software is called Learning Rate (also known as alpha). The learning rate setting can be controlled by moving a slider back and forth. On one side of the slider is the value 1E-05 on the other side is just 1. In between are various values such as 9E-05, .000045, etc. What I want to know is which one of these 2 learning rates is the fastest learning rate, 1E-05 on one side or 1 on the other. Thanks.

Answer 1

Learning rate is not about speed of training it is about size of the step when using quite naive approximation of the function (linear - for 1st order optimizers, or quadratic - for 2nd order). Consequently very small learning rate should lead to slow training, but big learning rate can lead to lack of training. Furthermore - values in between can still be not monotonic (you can have training where smaller learning rate actually converges faster than bigger one). So even though naively we could say that big learning rate is faster training - in general this is not true, furthermore - one cannot answer what learning rate is the fastest one. You can only use some general heuristics/observations here - you can start with big learning rate, and if results are bad, try reducing it. But in terms of actual training time guarantees - there are none.