Tensorflow不会分配完整的GPU内存

Question

Tensorflow allocates all of GPU memory per default, but my new settings actually only are 9588 MiB / 11264 MiB. I expected around 11.000MiB like my old settings.

Tensorflow information is here:

$ from tensorflow.python.client import device_lib
$ print(device_lib.list_local_devices())

[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 9709578925658430097
, name: "/gpu:0"
device_type: "GPU"
memory_limit: 9273834701
locality {
  bus_id: 1
}
incarnation: 16668416364446126258
physical_device_desc: "device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0"
, name: "/gpu:1"
device_type: "GPU"
memory_limit: 9273834701
locality {
  bus_id: 1
}
incarnation: 2094938711079475130
physical_device_desc: "device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:04:00.0"
]

nvidia-smi.exe says:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 385.41                 Driver Version: 385.41                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108... WDDM  | 00000000:03:00.0 Off |                  N/A |
| 23%   35C    P8    13W / 250W |   9284MiB / 11264MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108... WDDM  | 00000000:04:00.0 Off |                  N/A |
| 23%   38C    P2    55W / 250W |   9146MiB / 11264MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1280    C+G   ...mmersiveControlPanel\SystemSettings.exe N/A      |
|    0      1448      C   ...ers\Administrator\Anaconda3\pythonw.exe N/A      |
|    0      1560    C+G   Insufficient Permissions                   N/A      |
|    0      4120    C+G   ...6)\Google\Chrome\Application\chrome.exe N/A      |
|    0      4580    C+G   C:\Windows\explorer.exe                    N/A      |
|    0      5188    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0      5324    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    1      1228    C+G   Insufficient Permissions                   N/A      |
|    1      1244    C+G   Insufficient Permissions                   N/A      |
|    1      1448      C   ...ers\Administrator\Anaconda3\pythonw.exe N/A      |
+-----------------------------------------------------------------------------+

My environment is this:

OS: Windows10 library: python 3.6, keras 2.0.8, tensorflow-gpu 1.3.0, CUDA8.0 CUDNN6.0

Does anyone know the reason?

Answer 1

It is necessary to use the TCC driver to avoid windows reserving some of the VRAM. You may be using the WDDM driver.

Here is the page on TCC: https://docs.nvidia.com/gameworks/content/developertools/desktop/nsight/tesla_compute_cluster.htm

Here is a related question: How can I use 100% of VRAM on a secondary GPU from a single process on windows 10?

Answer 2

import tensorflow as tf
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.2)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))
from keras import backend as K
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.2
session = tf.Session(config=config)
K.set_session(session)

This works well for my case