Tensorflow-gpu with Keras Error

Question

Using Ubuntu 16.04, PyCharm

I used the following link to install tensorflow-gpu with python 3.5 : http://www.python36.com/install-tensorflow141-gpu/

Tensorflow installation is fine. It ran with test code. Then I installed Keras to run codes from here: https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.1-introduction-to-convnets.ipynb

I got the following error:

2018-02-23 11:19:13.457201: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2018-02-23 11:19:13.457535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7845 pciBusID: 0000:01:00.0 totalMemory: 5.93GiB freeMemory: 5.65GiB

2018-02-23 11:19:13.457551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)

2018-02-23 11:21:22.130004: E tensorflow/stream_executor/cuda/cuda_dnn.cc:378] Loaded runtime CuDNN library: 7005 (compatibility version 7000) but source was compiled with 6021 (compatibility version 6000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.

2018-02-23 11:21:22.130663: F tensorflow/core/kernels/conv_ops.cc:667] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)

Tensorflow-cpu works fine with Keras. My question is:

1) which instructions to follow to correctly install tensorflow-gpu with Keras for my setup?

2) What can I do eliminate these errors?

3) Is there any universal instructions that could be followed to correctly install tensorflow-gpu with Keras for all platforms?

Edit Here is a file I created which shows how to run Keras with tensorflow-gpu in Pycharm-community edition. https://github.com/mdkhan48/AgBot2018_VidStream/blob/master/how%20to%20run%20tensorflow-gpu.odt

Answer 1

There are 4 components you are trying to install:

1. Cuda
2. Cudnn
3. tensorflow
4. keras

they are not always synchronized, as of today tensorflow supports cuda 9.1 if you compile it yourself (like the guide you posted), the prebuilt binaries were compiled with cuda 9.0 and cudnn 7. I prefer using them so this is this is the cuda version I'm using.

this is how you can install it:

first, remove existing cuda installations:

sudo apt-get remove --purge nvidia-*

install ubuntu headers and cuda 9.0:

sudo apt-get install linux-headers-$(uname -r)
CUDA_REPO_PKG=cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
wget -O /tmp/${CUDA_REPO_PKG} http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/${CUDA_REPO_PKG}
sudo dpkg -i /tmp/${CUDA_REPO_PKG}
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
rm -f /tmp/${CUDA_REPO_PKG}
sudo apt-get update
sudo apt-get install -y cuda-9-0

download cuDNN v7.0.5 Library for Linux from nvidia site , navigate to downloaded folder and install it:

tar -xzvf cudnn-9.0-linux-x64-v7.tgz
sudo cp cuda/include/* /usr/local/cuda/include/
sudo cp cuda/lib64/* /usr/local/cuda/lib64/

reboot your machine and check nvidia driver works using:

nvidia-smi

the output should look like this:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.30                 Driver Version: 390.30                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  Off  | 00000000:04:00.0  On |                  N/A |
|  0%   33C    P8    N/A / 120W |    310MiB /  4038MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1307      G   /usr/lib/xorg/Xorg                           141MiB |
|    0      2684      G   compiz                                       120MiB |
|    0      4078      G   ...-token=                                    45MiB |
+-----------------------------------------------------------------------------+

create a virtual environment and install tensorflow and keras in it:

sudo apt-get install virtualenv
virtualenv tfenv
source tfenv/bin/activate
pip install tensorflow-gpu
pip install keras

Answer 2

That error is due to wrong version of cudnn. I think you have missed some steps in the tutorial. This error Loaded runtime CuDNN library: 7005 (compatibility version 7000) but source was compiled with 6021 (compatibility version 6000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration. Loaded runtime CuDNN library: 7005 (compatibility version 7000) but source was compiled with 6021 (compatibility version 6000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration. explains itself what is the problem.

My suggestion is to install cudnn 7.0.5 and if you have cuda 9.1 installed than build again from source and read carefully the instructions or if have cuda 9.0 installed than use pip3 install tensorflow-gpu .

Also i can conform that keras works with cuda 9.1 and tested the code successfully that you have tried and linked above.