Free GPU memory in Google Colab

Question

I wanted to know if there is a way to free GPU memory in Google Colab.

I am training some CNN in a loop with eurosat/rgb/ dataset from tf.datasets . The models are not that big, nor is the dataset.

The error is the following:

Epoch 1/8

---------------------------------------------------------------------------

ResourceExhaustedError                    Traceback (most recent call last)

<ipython-input-15-c4badfe8da7d> in <module>()
     27                           nclasses=NCLASSES,
     28                           metadic = METADIC,
---> 29                           val_split = 0.20)
     30     plot_results(record=current_exp,run='avg',batch=False,save=True)
     31     plot_results(record=current_exp,run='avg',batch=True,save=True)

7 frames

<ipython-input-6-f1fac48c4ac9> in run_experiment(bloques, input_shape, init_conv_filters, batch_size, epochs, init_lr, end_lr, nruns, optimizer, sma_periods, nclasses, metadic, val_split)
     75               epochs = epochs,
     76               workers = 1,
---> 77               callbacks = [LRFinder]
     78               )
     79 

/usr/local/lib/python3.7/dist-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1182                 _r=1):
   1183               callbacks.on_train_batch_begin(step)
-> 1184               tmp_logs = self.train_function(iterator)
   1185               if data_handler.should_sync:
   1186                 context.async_wait()

/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    883 
    884       with OptionalXlaContext(self._jit_compile):
--> 885         result = self._call(*args, **kwds)
    886 
    887       new_tracing_count = self.experimental_get_tracing_count()

/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    948         # Lifting succeeded, so variables are initialized and we can run the
    949         # stateless function.
--> 950         return self._stateless_fn(*args, **kwds)
    951     else:
    952       _, _, _, filtered_flat_args = \

/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
   3038        filtered_flat_args) = self._maybe_define_function(args, kwargs)
   3039     return graph_function._call_flat(
-> 3040         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
   3041 
   3042   @property

/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1962       # No tape is watching; skip to running the function.
   1963       return self._build_call_outputs(self._inference_function.call(
-> 1964           ctx, args, cancellation_manager=cancellation_manager))
   1965     forward_backward = self._select_forward_and_backward_functions(
   1966         args,

/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
    594               inputs=args,
    595               attrs=attrs,
--> 596               ctx=ctx)
    597         else:
    598           outputs = execute.execute_with_cancellation(

/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58     ctx.ensure_initialized()
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:
     62     if name is not None:

ResourceExhaustedError:  failed to allocate memory
     [[node dense1/kernel/Regularizer/Square (defined at <ipython-input-6-f1fac48c4ac9>:77) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
 [Op:__inference_train_function_309982]

Function call stack:
train_function

What I've tried so far:

I've done some research and I call the following function after each model is trained

def reset_tensorflow_keras_backend():
    # to be further investigated, but this seems to be enough
    import tensorflow as tf
    import tensorflow.keras as keras
    tf.keras.backend.clear_session()
    tf.compat.v1.reset_default_graph()
    _ = gc.collect()

As a way to get a fresh session for each model. I've been able to run a loop with several models and not getting the error, but today the error came up again in one of the simplest models, which is weird.

The GPU usage at the moment of failure is:

!nvidia-smi


+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.74       Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   48C    P0    57W / 149W |  11077MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Which is clearly near 100%.

Answer 1

This happened probably because every time you open a session in colab you don't get always the same GPU, you can check the GPU assigned like this.

!nvidia-smi -L  

What i do is reset the session until google bless me with a Tesla T4.

I searched in the past way to free the memory, but the only way is to restart the session.

I am confident that by picking the GPU you won't get the problem again.

As you can see google assigned you a Tesla K80 , the worst one possible