Tensorflow亚当优化器

Question

Okey I have been reading some of the posts regarding AdamOptimizer in tensorflow. I think there is some confusion around, at least among beginners in NNs like me.

If I understood correctly, tf.train.AdamOptimizer keeps a so-called "adaptative learning rate". I thought that this learning rate would grow smaller as time increases.

However, when I plot the function by which the learning rate is scaled, taken from the docs ,

t <- t + 1
lr_t <- learning_rate * sqrt(1 - beta2^t) / (1 - beta1^t)

this is what I get:

t = np.arange(200)
result = np.sqrt(1-0.999**t)/(1-0.9**t)
plt.plot(result)
plt.show

So, for t = 1, the value for the user-selected learning rate is multiplied by 0.3 Then it decreases quite fast until 0.15 of its value, and then increases with time, slowly approaching the limit = user-selected learning rate.

Isn't it a bit weird? I guess somewhere I am wrong, but I would've expected the learning rate to start at a higher value and then progressively decreasing towards smaller values.

Answer 1

You can not really plot the Adam learning rate like this, since Adam is a momentum optimizer. The applied gradient for each steps depends on a moving average of the mean and standard deviation of the gradients of previous steps.

In general there is no guarantee for the learning to converge, the raw learning rate alpha itself is not directly changed by Adams. It is only rescaled using the momentums of the gradient. The learning only converges well if mean and standard deviation of the gradient decrease over time when reaching the global minimum, which is often the case for simple neural networks.

For highly stochastic problems however one might still need to implement some form of learning rate decay to suppress 'oscillations' around the optimal parameters, or at least make them smaller to make sure there really is convergence.

If you really want to understand how exactly this works you might want to read the Adam paper , it is much simpler than it seems on first sight.