Tensorflow：关于adam优化器的困惑

Question

I'm confused regarding as to how the adam optimizer actually works in tensorflow.

The way I read the docs , it says that the learning rate is changed every gradient descent iteration.

But when I call the function I give it a learning rate. And I don't call the function to let's say, do one epoch (implicitly calling # iterations so as to go through my data training). I call the function for each batch explicitly like

for epoch in epochs
     for batch in data
          sess.run(train_adam_step, feed_dict={eta:1e-3})

So my eta cannot be changing. And I'm not passing a time variable in. Or is this some sort of generator type thing where upon session creation t is incremented each time I call the optimizer?

Assuming it is some generator type thing and the learning rate is being invisibly reduced: How could I get to run the adam optimizer without decaying the learning rate? It seems to me like RMSProp is basically the same, the only thing I'd have to do to make it equal (learning rate disregarded) is to change the hyperparameters momentum and decay to match beta1 and beta2 respectively. Is that correct?

Answer 1

I find the documentation quite clear, I will paste here the algorithm in pseudo-code:

Your parameters :

• learning_rate : between 1e-4 and 1e-2 is standard
• beta1 : 0.9 by default
• beta2 : 0.999 by default
• epsilon : 1e-08 by default

  The default value of 1e-8 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1.

---

Initialization:

m_0 <- 0 (Initialize initial 1st moment vector)
v_0 <- 0 (Initialize initial 2nd moment vector)
t <- 0 (Initialize timestep)

m_t and v_t will keep track of a moving average of the gradient and its square, for each parameters of the network. (So if you have 1M parameters, Adam will keep in memory 2M more parameters)

---

At each iteration t , and for each parameter of the model :

t <- t + 1
lr_t <- learning_rate * sqrt(1 - beta2^t) / (1 - beta1^t)

m_t <- beta1 * m_{t-1} + (1 - beta1) * gradient
v_t <- beta2 * v_{t-1} + (1 - beta2) * gradient ** 2
variable <- variable - lr_t * m_t / (sqrt(v_t) + epsilon)

---

Here lr_t is a bit different from learning_rate because for early iterations, the moving averages have not converged yet so we have to normalize by multiplying by sqrt(1 - beta2^t) / (1 - beta1^t) . When t is high ( t > 1./(1.-beta2) ), lr_t is almost equal to learning_rate

---

To answer your question, you just need to pass a fixed learning rate , keep beta1 and beta2 default values, maybe modify epsilon , and Adam will do the magic :)

---

Link with RMSProp

Adam with beta1=1 is equivalent to RMSProp with momentum=0 . The argument beta2 of Adam and the argument decay of RMSProp are the same.

However, RMSProp does not keep a moving average of the gradient. But it can maintain a momentum, like MomentumOptimizer.

A detailed description of rmsprop.

• maintain a moving (discounted) average of the square of gradients
• divide gradient by the root of this average
• (can maintain a momentum)

---

Here is the pseudo-code:

v_t <- decay * v_{t-1} + (1-decay) * gradient ** 2
mom = momentum * mom{t-1} + learning_rate * gradient / sqrt(v_t + epsilon)
variable <- variable - mom

Answer 2

RMS_PROP and ADAM both have adaptive learning rates .

The basic RMS_PROP

cache = decay_rate * cache + (1 - decay_rate) * dx**2
x += - learning_rate * dx / (np.sqrt(cache) + eps)

You can see originally this has two parameters decay_rate & eps

Then we can add a momentum to make our gradient more stable Then we can write

cache = decay_rate * cache + (1 - decay_rate) * dx**2
**m = beta1*m + (1-beta1)*dx**  [beta1 =momentum parameter in the doc ]
x += - learning_rate * dx / (np.sqrt(cache) + eps)

Now you can see here if we keep beta1 = o Then it's rms_prop without the momentum .

Then Basics of ADAM

In cs-231 Andrej Karpathy has initially described the adam like this

Adam is a recently proposed update that looks a bit like RMSProp with momentum

So yes ! Then what makes this difference from the rms_prop with momentum ?

m = beta1*m + (1-beta1)*dx
v = beta2*v + (1-beta2)*(dx**2)
**x += - learning_rate * m / (np.sqrt(v) + eps)**

He again mentioned in the updating equation m , v are more smooth .

So the difference from the rms_prop is the update is less noisy .

What makes this noise ?

Well in the initialization procedure we will initialize m and v as zero .

m=v=0

In order to reduce this initializing effect it's always to have some warm-up . So then equation is like

m = beta1*m + (1-beta1)*dx          beta1 -o.9 beta2-0.999
**mt = m / (1-beta1**t)**
v = beta2*v + (1-beta2)*(dx**2)
**vt = v / (1-beta2**t)**
x += - learning_rate * mt / (np.sqrt(vt) + eps)

Now we run this for few iterations . Clearly pay attention to the bold lines , you can see when t is increasing (iteration number) following thing happen to the mt ,

mt = m