CUDA_ERROR_OUT_OF_MEMORY tensorflow

Question

As part of my study project, I try to train a neural network which makes a segmentation on images (based on FCN), and during the execution I received the following error message:

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,67,1066,718] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Note that I have fixed the batch_size to 1 and I have the same error even when I tried different image sizes , I put also just 1 image to train instead of 1600 still the same error! Could you help me to solve this problem ? What is it really about ?

Answer 1

Welcome to Stackoverflow.

You should check your GPU and the available memory.

This is not a problem with tensorflow. But your graphicscard is too small.

You may have to use a network with lower memory requirements or a larger graphics card. It's not only about the size of the image you put in but all the weights need to be stored on the gpu too.

If you need to work with this specific net and you don't want to buy another pc you can use a CloudService such as AWS or GoogleCloudPlatform.

To check your GPU status you can use nvidia-smi