Keras不打印输出，高内存和CPU使用率，并且在使用张量板回调时不使用GPU

Question

I have a wierd situation in Keras and it really freaks me out. I am trying to train a CNN using pretrained Inception with additional convolution, global average pool and dense layers. I am using a ImageDataGenerator to load the data.

The data generator is working fine, I have tested that. The model compiles well also. But when I run fit_generator, no output is printed, the CPU is at 100% and memory starts filling up slowly until it overflows. And although I have a GPU and have worked with it in tensorflow (which is the backend here) a number of times, it is completely ignored by Keras.

Considering that maybe batch size could be a problem, I set it to 1 but it did not solve the issue. The images are of size 299x299, which is not that big anyway.

I will post the code below as a reference though it seems to me that nothing is wrong with it:

def get_datagen():
    return ImageDataGenerator(rotation_range=30,
                        width_shift_range=0.2,
                        height_shift_range=0.2,
                        horizontal_flip=True,
                        fill_mode='nearest'
                        )

# Setup and compile the model.
model = InceptionV3(include_top=False, input_shape=(None, None, 3))

# Set the model layers to be untrainable
for layer in model.layers:
    layer.trainable = False

x = model.output
x = Conv2D(120, 5, activation='relu')(x)
x = GlobalAveragePooling2D()(x)
predictions = Activation('softmax')(x)

model_final = Model(inputs=model.inputs, outputs=predictions)

model_final.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])

# Define the dataflow.
train_gen = get_datagen()
val_test_gen = get_datagen()

train_data = train_gen.flow_from_directory(train_folder, target_size=(299, 299), batch_size=1)
val_data = val_test_gen.flow_from_directory(validation_folder, target_size=(299, 299), batch_size=1)
test_data = val_test_gen.flow_from_directory(test_folder, target_size=(299, 299), batch_size=1)

train_size = train_data.n
val_size = val_data.n
test_size = test_data.n

# Define callbacks.
model_checkpoint = ModelCheckpoint('../models/dbc1/', monitor='val_accuracy', verbose=1, save_best_only=True)
early_stopping = EarlyStopping(monitor='val_accuracy', patience=3, verbose=1, mode='max')
tensorboard = TensorBoard(log_dir='../log/dbc1', histogram_freq=1, write_grads=True, )


model_final.fit_generator(train_data, steps_per_epoch=1, epochs=100, 
                          callbacks=[model_checkpoint, early_stopping, tensorboard],
                         validation_data=val_data, verbose=1)

EDIT

It seems the tensorboard callback was the problem here. When I remove it, everything works. Does anyone know why this is happening?

Answer 1

There's seems to be a problem (possibly related to keras#3358 ) when using the histogram_freq=1 under certain conditions.

You could try to set histogram_freq=0 and submit an issue at keras repository. You wouldn't have the gradient histograms, but at least you would be able to train:

model.fit(...,
          callbacks=[
              TensorBoard(log_dir='./logs/', batch_size=batch_size),
              ...
          ])

I notice that this problem doesn't happen with all trained models. If InceptionV3 usage is not a requirement, I recommend you switching to another model. So far, I found that the following code (adapted from yours, using VGG19 ) works on keras==2.1.2 , tensorflow==1.4.1 :

from keras.applications import VGG19
from keras.applications.vgg19 import preprocess_input

input_shape = (224, 224, 3)
batch_size = 1

model = VGG19(include_top=False, input_shape=input_shape)
for layer in model.layers:
    layer.trainable = False

x, y = model.input, model.output
y = Conv2D(2, 5, activation='relu')(y)
y = GlobalAveragePooling2D()(y)
y = Activation('softmax')(y)

model = Model(inputs=model.inputs, outputs=y)
model.compile('adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

g = ImageDataGenerator(rotation_range=30,
                       width_shift_range=0.2,
                       height_shift_range=0.2,
                       horizontal_flip=True,
                       preprocessing_function=preprocess_input)

train_data = g.flow_from_directory(train_folder,
                                   target_size=input_shape[:2],
                                   batch_size=batch_size)
val_data = g.flow_from_directory(validation_folder,
                                 target_size=input_shape[:2],
                                 batch_size=batch_size)
test_data = g.flow_from_directory(test_folder,
                                  target_size=input_shape[:2],
                                  batch_size=batch_size)

model.fit_generator(train_data, steps_per_epoch=1, epochs=100,
                    validation_data=val_data, verbose=1,
                    callbacks=[
                        ModelCheckpoint('./ckpt.hdf5',
                                        monitor='val_accuracy',
                                        verbose=1,
                                        save_best_only=True),
                        EarlyStopping(patience=3, verbose=1),
                        TensorBoard(log_dir='./logs/',
                                    batch_size=batch_size,
                                    histogram_freq=1,
                                    write_grads=True)])