Keras: how learning rate changes when Adadelta optimizer is used?

Question

For example I use Adadelta for optimizer when compile network model, then learning rate will change in time by this rule (but what is iterations ? ) and how can I log learning rate value to console?

model.compile(loss=keras.losses.mean_squared_error,
        optimizer= keras.optimizers.Adadelta())

In documentation lr is just starting learning rate?

Answer 1

The rule is related to updates with decay. Adadelta is an adaptive learning rate method which uses exponentially decaying average of gradients.

Answer 2

Looking at Keras source code , learning rate is recalculated based on decay like:

lr = self.lr
if self.initial_decay > 0:
    lr *= (1. / (1. + self.decay * K.cast(self.iterations, K.dtype(self.decay))))

So yes, lr is just starting learning rate.

To print it after every epoch, as @orabis mentioned, you can make a callback class:

class YourLearningRateTracker(Callback):
    def on_epoch_end(self, epoch, logs=None):
        lr = self.model.optimizer.lr
        decay = self.model.optimizer.decay
        iterations = self.model.optimizer.iterations
        lr_with_decay = lr / (1. + decay * K.cast(iterations, K.dtype(decay)))
        print(K.eval(lr_with_decay))

and then add its instance to the callbacks when calling model.fit() like:

model.fit(..., callbacks=[YourLearningRateTracker()])

However, note that, by default, decay parameter for Adadelta is zero and is not part of the “standard” arguments, so your learning rate would not be changing its value when using default arguments. I suspect that decay is not intended to be used with Adadelta.

On the other hand, rho parameter, which is nonzero by default, doesn't describe the decay of the learning rate, but corresponds to the fraction of gradient to keep at each time step (according to the Keras documentation ).

I found some relevant information on this Github issue , and by asking a similar question .