Keras Tensorboard Error when Histogram Frequency active

Question

I am running a simple Neural Network with Keras, backend Tensorflow, when trying to use Tesorboard to monitor training.

My model is the following:

import keras
from keras.layers.core import Dense, Activation, Dropout
from keras.models import Sequential
model = Sequential()
model.add(Dense(32, input_dim=500))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(2, activation='softmax'))
model.compile(optimizer='rmsprop',
          loss='binary_crossentropy',
          metrics=['accuracy'])

I try to run it using the following configuration:

tensorboard = keras.callbacks.TensorBoard(log_dir="../TFlogs",
                                          histogram_freq=2, batch_size=300, 
                                          write_grads=True, write_images=True)
one_hot_labels = keras.utils.to_categorical(y_train, num_classes=2)
history = model.fit(X_train, one_hot_labels, epochs=10,
                    validation_split=0.1, batch_size=300, callbacks=[tensorboard])

However for some reason I get the following error:

  File "C:/Users/user/PycharmProjects/SpamFilter/Filter/NeuralModel.py", line 108, in fit
history = model.fit(X_train, one_hot_labels, epochs=10, validation_split=0.1, batch_size=300, callbacks=[tensorboard])
  File "C:\Users\user\PycharmProjects\SpamFilter\venvKeras\lib\site-packages\keras\models.py", line 1002, in fit
validation_steps=validation_steps)
  File "C:\Users\user\PycharmProjects\SpamFilter\venvKeras\lib\site-packages\keras\engine\training.py", line 1705, in fit
validation_steps=validation_steps)
  File "C:\Users\user\PycharmProjects\SpamFilter\venvKeras\lib\site-packages\keras\engine\training.py", line 1256, in _fit_loop
callbacks.on_epoch_end(epoch, epoch_logs)
  File "C:\Users\user\PycharmProjects\SpamFilter\venvKeras\lib\site-packages\keras\callbacks.py", line 77, in on_epoch_end
callback.on_epoch_end(epoch, logs)
 File "C:\Users\user\PycharmProjects\SpamFilter\venvKeras\lib\site-packages\keras\callbacks.py", line 855, in on_epoch_end
result = self.sess.run([self.merged], feed_dict=feed_dict)
 File "C:\Users\user\PycharmProjects\SpamFilter\venvKeras\lib\site-packages\tensorflow\python\client\session.py", line 905, in run
run_metadata_ptr)
 File "C:\Users\user\PycharmProjects\SpamFilter\venvKeras\lib\site-packages\tensorflow\python\client\session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
 File "C:\Users\user\PycharmProjects\SpamFilter\venvKeras\lib\site-packages\numpy\core\numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.

When I set the TensorBoard option histogram_freq=0 I don't get the error, but I want to get the histogram information can someone help me find the reason for this?

Answer 1

Try running K.clear_session() before creating your model.

See this GitHub thread for reference to this error.

Answer 2

This may be related to K.learing_phase() . Especially if you have done K.set_learning_phase(1) before.

To diagnose: Run print(K.learning_phase()) , if it returns an int, then this problem is almost surely related to this issue. Try removing all sentences related to K.set_learing_phase(1) and see if it makes a difference.