Conda, Keras, cuDNN: different versions showing

Question

I'm using Anaconda (in Ubuntu 18.04) and I have an environment with Keras (and tensorflow-gpu) installed. Here are the different versions:

• Keras: 2.2.4
• Tensorflow-GPU: 1.15.0
• CuDNN: 7.6.5 for Cuda10.0.0
• CudaToolKit: 10.0.130

The version are chosen by Conda, but I'm wondering why it downloaded 10.0 when nvidia-smi shows me that my cuda should be (or is?) 10.1:

NVIDIA-SMI 435.21 Driver Version: 435.21 CUDA Version: 10.1

But, fun fact, when I do nvcc --version:

Cuda compilation tools, release 9.1, V9.1.85

So here comes my question(s): what version of Cuda am I using? What version of Cuda should I be using? Does Anaconda handle Cuda by environment?

PS: (this is not my question, but why I ask it)

I'm asking that because I'm running into this issue:

tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

I looked for an solution ( could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR ) but none of the answer I tried worked (deleting files, running in sudo, etc) so I think it's a compatibility issue

Answer 1

Note: altough I do not consider this answer as THE solution, it allowed me to continue working on my project so it's good enough for the moment.

1. Reinstall Cuda 10.1 (not 10.2 in my case because of some issue with the driver 440 with Steam) (check what version your nvidia driver is and be sure to install the correct Cuda for that version)
2. Follow the post-installation: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions
3. Use whereis cuda to find if there are other version left on the system (in my case, I had cuda-dev-9.1, which explains why nvcc -V showed that version)
4. Delete all old versions
5. Normally, nvcc -V and nvidia-smi should show the same Cuda version
6. Reinstall cudnn if needed

Now, this doesn't fix the bug:

Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

A working solution (but still not awesome) is to add the following code on top of your python file (I use Keras, but it works with TensorFlow alone as well):

from keras.backend.tensorflow_backend import set_session
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
set_session(sess)

And it's (apparently) working!

Big thanks to Berkay of his support!

(technically, try to delete the old versions before adding another one, but it works too)