How can I print intermediate states for a variation of a Keras' SGD optimizer when using Tensorflow backend

Question

I'd like to write a variant of Keras' SGD, that allows for a discrete change in step size at specified iterations. I'm using a Tensorflow backend.

To help debug, I'm trying to make the get_updates method of the optimizer print messages to me, but I can't seem to do it. I've tried both standard print statements and tf.Print, but neither works. The relevant code, which comes straight from Keras SGD optimizer cldass looks like this:

@interfaces.legacy_get_updates_support
def get_updates(self, loss, params):
    print (" -------------------------> Getting updates <------------------------------------------")
    grads = self.get_gradients(loss, params)
    self.updates = [K.update_add(self.iterations, 1)]
    tf.Print(self.iterations,
             [self.iterations],
             message="-------------------------------> GETTING UPDATES <----------------------------------------")

    lr = self.lr
    if self.initial_decay > 0:
        lr = lr * (1. / (1. + self.decay * K.cast(self.iterations,
                                                  K.dtype(self.decay))))

    # momentum
    shapes = [K.int_shape(p) for p in params]
    moments = [K.zeros(shape) for shape in shapes]
    self.weights = [self.iterations] + moments
    for p, g, m in zip(params, grads, moments):
        v = self.momentum * m - lr * g  # velocity
        self.updates.append(K.update(m, v))

        if self.nesterov:
            new_p = p + self.momentum * v - lr * g
        else:
            new_p = p + v

        # Apply constraints.
        if getattr(p, 'constraint', None) is not None:
            new_p = p.constraint(new_p)

        self.updates.append(K.update(p, new_p))
    return self.updates

Although the print statement does manage to relay one message, that is all. I expected to see output each time parameters were updated (ie after each batch). Instead, I only see the printed output just prior to the first epoch of training.

What am I doing wrong? Am I still tied up in the difficulties (for me) of working with graph calculations? Also, shouldn't tf.Print have produced some text output?

Answer 1

I think I know what's going on here.....

1. My print statement only produces output when get_updates is called. It is called only once and it returns the graph (sub-graph?) that is used to actually compute updates.

2. My tf.Print produces no output, because I never explicitly put it in the computational graph

Answer 2

In order to use tf.print in graph mode, you can just use tf.print as a drop-in replacement for tf.Print , you just have to force the execution of the tf.print operation before the execution of a tensor in your model function . You can check here for details.