当在 keras 层中指定 `input_shape` 时，Tensorflow 2.0 会泄漏内存

Question

I'm using Tensorflow 2.0 with keras APIs. There are situations where I need to generate multiple models within the same program execution (say, a cross-validation type of situation). When I generate these models without specifying the input shape, as in the following code snippet, there is no memory leak.

import time

from tensorflow import keras


for _ in range(100):
    model = keras.model.Sequential()
    model.add(keras.kayers.Dense(120, activation='relu'))
    model.compile(loss='binary_crossentropy',
                  optimizer=keras.optimizers.SGD())
    time.sleep(0.1)

However, if I specify the input shape when adding the first layer of the model, it seems like models are piling up in memory, without being destroyed when not needed. It also seems like the execution time increases quite a lot.

import time

from tensorflow import keras


for _ in range(100):
    model = keras.model.Sequential()
    model.add(keras.kayers.Dense(120, activation='relu', input_shape=(10, 10)))
    model.compile(loss='binary_crossentropy',
                  optimizer=keras.optimizers.SGD())
    time.sleep(0.1)

I also generated a list of all the objects for the 2 code snippets, using Pympler .

from pympler import muppy
from pympler import summary


...

all_objects = muppy.get_objects()
occupancy = summary.summarize(all_objects)
summary.print_(occupancy)

For the models generated without input shape, here is the result:

                                                                   types |   # objects |   total size
======================================================================== | =========== | ============
                                                             <class 'str |       79278 |     14.02 MB
                                                            <class 'dict |       14468 |      6.93 MB
                                                            <class 'code |       25252 |      3.49 MB
                                                            <class 'type |        2582 |      2.57 MB
                                                            <class 'list |        8573 |    944.06 KB
                                                           <class 'tuple |       12079 |    796.21 KB
                                                             <class 'set |         732 |    466.12 KB
                                                         <class 'weakref |        4475 |    349.61 KB
                                                     <class 'abc.ABCMeta |         341 |    346.27 KB
                     <class 'tensorflow.core.framework.op_def_pb2.ArgDef |        3822 |    328.45 KB
  <class 'google.protobuf.pyext.cpp_message.GeneratedProtocolMessageType |         369 |    320.16 KB
                                                            <class 'cell |        5921 |    277.55 KB
                                                     function (__init__) |        1694 |    224.98 KB
                                                        <class 'property |        2466 |    192.66 KB
                                              <class 'wrapper_descriptor |        2353 |    183.83 KB

And here is the snapshot for the models generated with the input shape.

                                                           types |   # objects |   total size
================================================================ | =========== | ============
                                                   <class 'tuple |      296304 |     26.77 MB
                                                    <class 'dict |       71296 |     16.14 MB
                                                     <class 'str |       84828 |     14.54 MB
                                                     <class 'int |      395032 |     10.55 MB
                                                    <class 'list |       98027 |      9.89 MB
                                                    <class 'code |       25281 |      3.49 MB
                                                    <class 'type |        2587 |      2.58 MB
                                                     <class 'set |        2944 |    975.00 KB
               <class 'tensorflow.python.framework.ops.Operation |       14100 |    771.09 KB
    <class 'tensorflow.python.framework.ops.Operation._InputList |       14100 |    771.09 KB
                  <class 'tensorflow.python.framework.ops.Tensor |       14100 |    771.09 KB
  <class 'tensorflow.python.pywrap_tensorflow_internal.TF_Output |       13200 |    721.88 KB
                                            <class 'SwigPyObject |       14100 |    660.94 KB
                                 <class 'collections.OrderedDict |        1262 |    622.22 KB
                                                 <class 'weakref |        6127 |    478.67 KB

Is there anything wrong I'm doing? According with the relevant keras documentation , the input shape should be somehow provided to the layer (the Tensorflow documentation about input shape says that if the layer has multiple inputs, the shape is not needed), however, my code still works fine without specifying the input shape. I'm afraid this might be a bug in Tensorflow 2, but honestly I don't really know what's wrong.

Notes

• Memory graphs are generated with the memory_profiler package.
• Memory snapshots are generated with the Pympler package.
• Python 3.7.1 is used.
• OS is Linux Elementary (Ubuntu).

Thank you for your support.

Answer 1

This is a long time after the post, but I found that my massive memory leak with TensorFlow.Keras was solved by just adding the line:

tf.keras.backend.clear_session()

To the start of each iteration. TensorFlow 2.0, Windows, Python 3.6/Anaconda