Keras: how to reset optimizer state?

Question

How to reset optimizer state in keras?

Looking at Optimizer class I can't see such a method: https://github.com/keras-team/keras/blob/613aeff37a721450d94906df1a3f3cc51e2299d4/keras/optimizers.py#L60

Also what is actually self.updates and self.weights ?

Answer 1

There isn't an "easy" way to reset the "states", but you can always simply recompile your model with a new optimizer (model's weights are preserved):

newOptimizer = Adadelta()
model.compile(optimizer=newOptimizer)     

You can also use the method set_weights(weightsListInNumpy) (not recommended), in the base class Optimizer , but this would be rather cumbersome as you would need to know all initial values and shapes, which sometimes may not be trivial zeroes .

Now, the property self.weights doesn't do much, but the functions that save and load optimizers will save and load this property. It's a list of tensors and should not be changed directly. At most use K.set_value(...) in each entry of the list. You can see the weights in saving the optimizer in the _serialize_model method .

The self.updates are something a little more complex to understand. It stores the variables that will be updated with every batch that is processed by the model in training. But it's a symbolic graph variable.

The self.updates , as you can see in the code, is always appended with a K.update(var, value) or K.update_add(var, value) . This is the correct way to tell the graph that these values should be updated every iteration.

Usually, the updated vars are iterations , params (the model's weights), moments , accumulators , etc.

Answer 2

I don't think there is a universal method for this, but you should be able to reset the state of your optimizer by initializing the variables holding it. This would need to be done with the TensorFlow API, though. The state variables depend on the specific kind of optimizer. For example, if you have a Adam optimizer ( source ), you could do the following:

from keras.optimizers import Adam
from keras import backend as K

optimizer = Adam(...)
# These depend on the optimizer class
optimizer_state = [optimizer.iterations, optimizer.lr, optimizer.beta_1,
                   optimizer.beta_2, optimizer.decay]
optimizer_reset = tf.variables_initializer(optimizer_state)

# Later when you want to reset the optimizer
K.get_session().run(optimizer_reset)

Answer 3

The optimizer is just adjusting the wheihts of your model, thus the information is stored in the model, not in the optimizer.

That means you can't reset an optimizer in a way you might think. You need to reset (or maybe easyier, recreate) your model.

That means you also can optimize your model with an optimizer A, stop after some epochs, and continue optimizing your model with optimizer B not loosing the progress optimizer A made allready.

I don't know exactly what self.updates and self.weights are there for. But because those are internal variables of the class someone needs to know/read about the optimizer class itself and understand its code. Here we need to wait fore someone who dived deeper into the sourcecode of keras.

EDIT

You can just recreate your optimizer for example:

model = Seqeuential()
...
...
...

model.compile(optimizer=keras.optimizers.Adadelta(lr = 5, loss='mean_squared_error')
model.fit(X, y, epochs=10)

model.compile(optimizer=keras.optimizers.Adadelta(lr = 0.5, loss='mean_squared_error')
model.fit(X, y, epochs=10)

With the above code you train 10 epochs with learning rate 5, compile your model with a new optimizer, and continue for another 10 epochs with learning rate 0.5. The weights which you could also call your training progress do not get lost if you compile your model again.