Print layer outputs in Keras during training

Question

I am new to Keras. How can I print the outputs of a layer, both intermediate or final, during the training phase?

I am trying to debug my neural network and wanted to know how the layers behave during training. To do so I am trying to exact input and output of a layer during training, for every step.

The FAQ ( https://keras.io/getting-started/faq/#how-can-i-obtain-the-output-of-an-intermediate-layer ) has a method to extract output of intermediate layer for building another model but that is not what I want. I don't need to use the intermediate layer output as input to other layer, I just need to print their values out and perhaps graph/chart/visualize it.

I am using Keras 2.1.4

Answer 1

I think I have found an answer myself, although not strictly accomplished by Keras.

Basically, to access layer output during training, one needs to modify the computation graph by adding a print node.

A more detailed description can be found in this StackOverflow question:
How can I print the intermediate variables in the loss function in TensorFlow and Keras?

I will quote an example here, say you would like to have your loss get printed per step, you need to set your custom loss function as:

for Theano backend:

diff = y_pred - y_true
diff = theano.printing.Print('shape of diff', attrs=['shape'])(diff)
return K.square(diff)

for Tensorflow backend:

diff = y_pred - y_true
diff = tf.Print(diff, [tf.shape(diff)])
return K.square(diff)

Outputs of other layers can be accessed similarly.

There is also a nice vice tutorial about using tf.Print() from Google
Using tf.Print() in TensorFlow

Answer 2

If you want to know more info on each neuron, you need to use the following to get their bias and weights.

weights = model.layers[0].get_weights()[0]
biases = model.layers[0].get_weights()[1]

0 index defines weights and 1 defines the bias.

You can also get per layer too,

for layer in model.layers:
    weights = layer.get_weights() # list of numpy arrays

After each training, if you can access each layer with its dimension and obtain the weights and bias to a numpy array, you should be able to visualize how the neuron after each training.

Hope it helps.