Tensorflow 在每次调用 session.run() 时都会泄漏内存并带有最终图

Question

I am trying to use the tf.data api to feed variable-size image data (LxLx2) to my model, however I notice that I am leaking memory at every iteration. I would expect that the memory use would be determined by the largest image in the dataset, however I can see that memory use is increasing even when processing an image that is smaller than the maximum size seen so far.

Leaking memory over 100 iterations

When I directly gather the processed features instead of computing the neural network activations, the memory does not seem to leak.

Expected memory use (forgoing NN computation)

It seems that the most common cause of this type of problem is dynamically adding nodes to the graph, however I call graph.finalize() prior to the iteration and do not catch any error.

I am using python 3.5.4 and tensorflow 1.10 and running the computation on the CPU only.

import tensorflow as tf
from sys import argv

# Data preparation
def record_parser(value):
    keys_to_features = {
        'seq_length': tf.VarLenFeature(dtype=tf.int64),
        'seq_feat': tf.VarLenFeature(dtype=tf.float32)
        }
    parsed = tf.parse_single_example(value, keys_to_features)
    length_ = tf.reshape(parsed['seq_length'].values, [])
    i32_len = tf.cast(length_, dtype=tf.int32)
    features_ = tf.reshape(parsed['seq_feat'].values, [i32_len, i32_len, 2])
    return features_

graph = tf.get_default_graph()
dataset_ = tf.data.TFRecordDataset(argv[1])
dataset_ = dataset_.map(lambda value: record_parser(value))
dataset_ = dataset_.batch(1)
iterator = dataset_.make_one_shot_iterator()
features = iterator.get_next()

# NN part
nn0 = tf.layers.conv2d(features, filters=64, kernel_size=15, padding='SAME',\
 activation=tf.nn.relu)
nn = tf.layers.dense(nn0, units=100, activation=tf.nn.relu)
prediction = tf.layers.dense(nn, 17, activation=None)

var_init_op = tf.group(
                tf.global_variables_initializer(),
                tf.local_variables_initializer()
                )
graph.finalize()

# Iterating over samples
with tf.Session() as sess:
    sess.run(var_init_op)
    for i in range(100):
        out_loss = sess.run(prediction)
        #out_loss = sess.run(features)

Answer 1

Mentioning the Answer here for the benefit of the Community.

The Issue of Memory Leakage when using tf.data API in Tensorflow Version 1.10 is resolved by upgrading to Tensorflow Version 1.13 .