High GPU Memory-Usage but zero volatile gpu-util

Question

My new training code occupied high GPU Memory-Usage but zero volatile gpu-util

/home/diana/data/KaggleDiabeticRetinopaI tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with     properties: 
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.835
pciBusID 0000:05:00.0
Total memory: 7.92GiB
Free memory: 7.81GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlowdevice (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:05:00.0)

I'm trying a new data generator, and the old one run with volatile gpu-util of 10%, so the environment or the tensorflow version won't be the key problem

so I wonder if anyone could tell me what element in the code could possibly cause this problem?

thank you very much！

Answer 1

Tensorflow by default pre-allocates all the available VRAM when you create a session, whether it is actually needed or not. 0% volatile GPU-util only shows that nothing is actually "running" on the GPUs(no active kernels).

If you don't want this to happen, you can set allow_growth to True and pass it appropriately in a config object when you create your session.

Read tf.Session and config.proto for more details.

EDIT: This is how you set allow_growth to True

session = tf.Session(config=tf.ConfigProto(gpu_options=tf.GPUOptions(allow_growth=True)))