Theano for GPU out of memory error

Question

A few hours ago, I got the Theano+Keras GPU environment up and running successfully. I even tested some code to make sure that it was being executed on the GPU. However, when I run import theano now, I get the following error:

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): . . . . GpuArrayException: cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory

I use a GPU on our university server and it is shared by many students in the lab. Is the error possibly due to insufficient memory due to other running processes? The output of nvidia-smi is shown below. Process with PID 29586 is mine.

+-----------------------------------------------------------------------------+ | NVIDIA-SMI 387.12 Driver Version: 387.12 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX TIT... Off | 00000000:01:00.0 On | N/A | | 48% 83C P2 196W / 250W | 12152MiB / 12198MiB | 99% Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     10977      C   python                                      5506MiB |
|    0     24129      C   python                                      6323MiB |
|    0     25238      G   /usr/lib/xorg/Xorg                           110MiB |
|    0     25773      G   /usr/bin/gnome-shell                          90MiB |
|    0     29586      C   python                                       106MiB |
+-----------------------------------------------------------------------------+

The GPU is an Nvidia Titan X. I have googled this error extensively and have tried so many methods over the past few hours. Please help.

Answer 1

To keep it simple, yes, the card runs out of memory. TITAN X has 12 GB of RAM and the first processes almost use all of it. Maybe ask the owner if they could stall their process or use a smaller batch size if they use it for Deep Learning.