Why does the CNN model still exceed the GPU memory even using the delete command in TensorFlow?

Question

I am just using TensorFlow to realise a CNN model. During the training process, there is an intermediate variable which occupies a large GPU memory and I want to clear the memory of this variable.

This variable is called 'rgb_concat', I just tried to use 'del rgb_concat', but the model still exceeds the GPU memory. I am not sure if using 'del' command could release the GPU memory? May I ask if I need to combine other commands with 'del' to release the GPU memory of this 'rgb_concat'?

Many thanks in advance!

An intermediate variable called 'rgb_concat' which occupies a large GPU memory and I want to clear it and save GPU memory for other layers in a CNN model. It seems that only using 'del' command could not release memory, I am not sure if it works and how to release memory.

x = input_image
for j in range(n_sub_layers):
    nn = Conv2dLayer(x, j)     #
    rgb_concat.append(nn)
    x = nn
rgb_concat_sublayer = ConcatLayer([rgb_concat[0], rgb_concat[1]], 
concat_dim=3, name='rgb_concat_sublayer_{}_{}'.format(i,1))
for sub_layer in range(2, n_sub_layers): #Second 'for' loop!!!
    rgb_concat_sublayer = ConcatLayer([rgb_concat_sublayer, 
    rgb_concat[sub_layer]], concat_dim=3, 
    name='rgb_concat_sublayer_{}_{}'.format(i,sub_layer))
del rgb_concat #try to use 'del' to delete variable 'rgb_concat' after the second 'for' loop

Answer 1

The way tensorflow works is by making graphs in the memory (RAM/GPU memory if you will). So deleting a python object is not going to help which only clear the memory used by that python stack memory, not the graph which is already made.

The memory occupied is only by the Conv2dLayer() part of the code, (the part of the graph), you are reusing it in you stack memory (python list rgb_concat ). ConcatLayer is also resuing all the already build graph locations, so creates minimal overhead. It's possible your n_sub_layers is large, which exceeds gpu memory. But can you paste your entire code with the values?