Using CUDA8 in theano

Question

I have working installtion of CUDA8 and have installed theano, while importing the theano it searches for CUDA7.5 instead of CUDA8, How can tell theano to use CUDA8 instead of CUDA7.5?

My sytem only have CUDA8, and it doesn't contain mixed environment cuda(ie having both CUDA7.5 and CUDA8).

Here is a output of nvidia-smi

$ nvidia-smi 
Sat Feb  4 11:32:30 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26                 Driver Version: 375.26                         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970M    Off  | 0000:01:00.0     Off |                  N/A |
| N/A   54C    P0    22W /  N/A |      0MiB /  3016MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |

Here is a output of nvcc -V

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Sep__4_22:14:01_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44

While importing theano in ipython it fails to run in gpu mode with error that, it can't find the libcudart.so.7.5

Python 3.6.0 (default, Jan 16 2017, 12:12:55) 
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import theano
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: libcudart.so.7.5: cannot open shared object file: No such file or directory
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu0 is not available  (error: cuda unavailable)

Here is a content of my .theanorc

[global]                                                                                                                                                                                                       
floatX = float32
device = gpu0
cuda.root = /opt/cuda

I tried to build theano from source, after uninstalling the previous installtion of it, that too is not working. I did cleared the theano-cache with theano-cache clean / theano-cache purge and my manually deleting the content under .theano directory, which too couldn't helped.

With more debugging I get error here https://github.com/Theano/Theano/blob/8b9f73365e4932f1c005a0a37b907d28985fbc5f/theano/gof/cmodule.py#L302

when nvcc_compiler tries to load the cuda_ndarray.so from cuda_ndarray in theano cache

comiplation phase for mod.cu runs without error.

In this case linker is pointing to wrong libcudart

readelf -a cuda_ndarray.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libcublas.so.8.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpython3.6m.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcudart.so.7.5]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

I assume ldconfig is properly caching the cuda libraries

$ sudo ldconfig -v | grep -e 'cuda\|blas'
/opt/cuda/lib64:
    libcublas.so.8.0 -> libcublas.so.8.0.45
    libcudart.so.8.0 -> libcudart.so.8.0.44
    libnvblas.so.8.0 -> libnvblas.so.8.0.44
/opt/cuda/nvvm/lib64:
    libcuda.so.1 -> libcuda.so.375.26
    libblas.so.3 -> libblas.so.3.7.0
    libicudata.so.58 -> libicudata.so.58.2
    libopenblas.so.0 -> libopenblas.so
    libicudata.so.58 -> libicudata.so.58.1

Answer 1

在进一步挖掘我的问题之后，我重构了我原来的问题并在这里发布了nvcc正在挑选错误的libcudart库来解决我的问题。