TensorFlow Python 脚本被杀死

Question

I am a beginner in TensorFlow. My TensorFlow script abruptly exits saying Killed . My code is as follows:

import tensorflow as tf
# Load data X_train, y_train and X_valid, y_valid

# An image augmentation pipeline
def augment(x):
    x = tf.image.random_brightness(x, max_delta=0.2)
    x = tf.image.random_contrast(x, 0.5, 2)
    return x

from sklearn.utils import shuffle
X_train, y_train = shuffle(X_train, y_train)

def LeNet(x):
    # Define LeNet architecture
    return logits

# Features:
x = tf.placeholder(tf.float32, (None, 32, 32, 3))
# Labels:
y = tf.placeholder(tf.int32, (None))
# Dropout probability
prob = tf.placeholder(tf.float32, (None))
# Learning rate
rate = tf.placeholder(tf.float32, (None))
rate_summary = tf.summary.scalar('learning rate', rate)

logits = LeNet(x)
accuracy_operation = # defined accuracy_operation

accuracy_summary = tf.summary.scalar('validation accuracy', accuracy_operation)
saver = tf.train.Saver()

summary = tf.summary.merge_all()
writer = tf.summary.FileWriter('./summary', tf.get_default_graph())

def evaluate(X_data, y_data):
    # Return accuracy with X_data, y_data
    return accuracy

with tf.Session() as sess:

    saver.restore(sess, './lenet')

    for i in range(EPOCHS):
        X_train, y_train = shuffle(X_train, y_train)
        for offset in range(0, len(X_train), BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset:end], y_train[offset:end]
            batch_x = sess.run(augment(batch_x))

            # Run the training operation, update learning rate

        validation_accuracy = evaluate(X_valid, y_valid)
        writer.add_summary(sess.run(summary, feed_dict = {x: X_valid, y: y_valid, prob: 1., rate: alpha}))

I have omitted the parts which I know for sure are not causing problems. I know which parts are fine because the script was not giving any troubles earlier. After adding certain parts (mainly the summary writer operations ), the script abruptly says Killed and exits after executing a certain number of training operations. I suspect this is due to a memory leak but I can't detect it.

Answer 1

I ran into a similar problem just a few days ago. In my case I had some operations that turned out to be computationally very heavy, as I learnt later. As soon as I reduced the size of my tensors, the message disappeared and my code ran. I can't tell exactly what is the cause of the issue in your case, but from my experience and from what you say (that error only comes up when adding summary) I would suggest to fiddle with the size of your X_valid, Y_valid. It might just be that the writer can't cope with too much data...