nvidia cuda and insufficient driver version

Question

I'm trying to use CUDA technology, but have some issues

greymachine ~/NVIDIA_CUDA-5.0_Samples/1_Utilities/deviceQuery $ ./deviceQuery 
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version

greymachine ~/NVIDIA_CUDA-5.0_Samples/1_Utilities/deviceQuery $

That's my problem.

My config:

$ nvidia-settings -q NvidiaDriverVersion
  Attribute 'NvidiaDriverVersion' (greymachine.localdomain:0.0): 310.19

$ uname -r
3.7.1-un-def-alt2.1

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2012 NVIDIA Corporation
Built on Fri_Sep_21_17:28:58_PDT_2012
Cuda compilation tools, release 5.0, V0.2.1221 





$ ./deviceQueryDrv 
./deviceQueryDrv Starting...

CUDA Device Query (Driver API) statically linked version 
Detected 1 CUDA Capable device(s)

Device 0: "GeForce GT 430"
  CUDA Driver Version:                           5.0
  CUDA Capability Major/Minor version number:    2.1
  Total amount of global memory:                 1024 MBytes (1073283072 bytes)
  ( 2) Multiprocessors x ( 48) CUDA Cores/MP:    96 CUDA Cores
  GPU Clock rate:                                1400 MHz (1.40 GHz)
  Memory Clock rate:                             800 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 131072 bytes
  Max Texture Dimension Sizes                    1D=(65536) 2D=(65536,65535) 3D=(2048,2048,2048)
  Max Layered Texture Size (dim) x layers        1D=(16384) x 2048, 2D=(16384,16384) x 2048
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 32768
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:           1024
  Maximum sizes of each dimension of a block:    1024 x 1024 x 64
  Maximum sizes of each dimension of a grid:     65535 x 65535 x 65535
  Texture alignment:                             512 bytes
  Maximum memory pitch:                          2147483647 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Concurrent kernel execution:                   Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Bus ID / PCI location ID:           1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > 





$ lsmod | grep nvidia
nvidia               9381500  39 
i2c_core               30993  3 i2c_i801,nvidia,videodev


dmesg:
[   28.548939] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  310.19  Thu Nov  8 00:52:03 PST 2012
[   29.065356] NVRM: GPU at 0000:01:00: GPU-5a5ce500-f7fd-ab9d-64d7-cc0d1fe26ff1
[   29.065360] NVRM: Your system is not currently configured to drive a VGA console
[   29.065361] NVRM: on the primary VGA device. The NVIDIA Linux graphics driver
[   29.065362] NVRM: requires the use of a text-mode VGA console. Use of other console
[   29.065363] NVRM: drivers including, but not limited to, vesafb, may result in
[   29.065364] NVRM: corruption and stability problems, and is not supported.
 1682.331776] NVRM: GPU at 0000:01:00: GPU-5a5ce500-f7fd-ab9d-64d7-cc0d1fe26ff1

$ ldd ./deviceQuery 
        linux-vdso.so.1 (0x00007fffd4dff000)
        libcudart.so.5.0 => /usr/local/cuda-5.0/lib64/libcudart.so.5.0 (0x00007f5e90c26000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f5e90922000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5e9070c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5e90362000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5e90145000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5e8ff40000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f5e8fd38000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f5e8fa3e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5e90eaa000) 

I've installed cuda toolkit, downloaded from the nvidia site, i've precompiled drivers from my distro (Alt Linux), but libcuda.so doesn't come with them, so i copied that library from original nvidia drivers. Compiling is ok. I've also tested 2.6.32 kernel with 304.51: got the same msg, but that is understandable, cuda comes with 304.54 driver.

AFAIU, if i have newer drivers, than came with cuda toolkit, that is ok. But as u see some thing is wrong.

So, can kernel drivers be newer, than original (ie drivers i got with cuda) May be i should compile modules myself? but what for, why? my distro modules work good.

Thanks

Answer 1

This appears to have been caused by a very broken CUDA installation. The usual solution for is to uninstall everything, install a supported host toolchain and then reinstall the driver and toolkit. Every toolkit contains detailed installation instructions and system requirements. If you read and follow these, there will rarely be an issue of this type.

[This answer was assembled from comments and added as a community wiki entry to get it off the unanswered question list]