Kebuntu上的keras（-gpu）+ tensorflow-gpu + anaconda

Question

I have Kubuntu 18.04 and Anaconda 5.2 64. I installed the CUDA drivers and keras-gpu and tensorflow-gpu (automatically also installed tensorflow).

The following code

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K

import tensorflow as tf
print('Tensorflow: ', tf.__version__)

gives the output

2018-07-29 12:14:06.821996: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-07-29 12:14:06.880569: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-07-29 12:14:06.880910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: GeForce GTX 980 major: 5 minor: 2 memoryClockRate(GHz): 1.2155
pciBusID: 0000:01:00.0
totalMemory: 3.95GiB freeMemory: 2.72GiB
2018-07-29 12:14:06.880924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2018-07-29 12:14:07.058984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-29 12:14:07.059012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958]      0
2018-07-29 12:14:07.059017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   N
2018-07-29 12:14:07.059114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/device:GPU:0 with 2430 MB memory) -> physical GPU (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0, compute capability: 5.2)
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 18195666940796676435
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 2548367360
locality {
  bus_id: 1
  links {
  }
}
incarnation: 7016427886680347829
physical_device_desc: "device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0, compute capability: 5.2"
]
Using TensorFlow backend.
Tensorflow:  1.9.0

So it seems keras is using tensorflow CPU instead of GPU (using DeepBach, one of my CPU cores is at 100%)? What am I doing wrong?

How can I find out which device is being used by keras/DeepBach? nvidia-smi shows no GPU utilisation during training with keras/DeepBach. How can I tell keras/DeepBach to use the GPU instead of the CPU?

CUDA seems to be installed:

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

$ nvidia-smi
Sun Jul 29 12:10:28 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.48                 Driver Version: 390.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980     Off  | 00000000:01:00.0  On |                  N/A |
|  4%   62C    P0    47W / 180W |   1160MiB /  4040MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1113      G   /usr/lib/xorg/Xorg                           562MiB |
|    0      1385      G   kwin_x11                                     152MiB |
|    0      1395      G   /usr/bin/krunner                               2MiB |
|    0      1399      G   /usr/bin/plasmashell                         167MiB |
|    0     26801      G   ...-token=2DD4BBFEA86302FEC3C179E07D55C897   267MiB |
+-----------------------------------------------------------------------------+

Answer 1

While you have your code running check system-monitor to see if GPU is involved or not. Check specifically for Gpu's memory usage

Answer 2

I think that you have compiled (or you installed already compiled package) tensorflow with CUDA support, but not with support all instructions available for your CPU (your CPU supports SSE4.1, SSE4.2, AVX, AVX2 and FMA instructions that tensorflow can use).

This means, that tensorflow will work fine (with full GPU support), but you can't use your processor at full capacity.

Try compare time (GPU vs CPU) with this example: https://stackoverflow.com/a/54661896/10418812