“CUDA driver version is insufficient for CUDA runtime version” - but it isn't

Question

I have CUDA 10.1 installed on my Linux system under under /usr/local/cuda-10.1 (and linked to from /usr/local/cuda ). After some package updates and removals (which I can go into if you like), I also have the NVIDIA drivers, version 440.82, installed (nvidia-smi says as much).

However, when I build and run this minimal program:

#include <cuda_runtime.h>
#include <cstdio>
#include <cstdlib>

int main() {
    int deviceCount;
    cudaError_t error_id = cudaGetDeviceCount(&deviceCount);
    if (error_id != cudaSuccess) {
        printf("cudaGetDeviceCount returned %d: %s\n", (int)error_id, cudaGetErrorString(error_id));
        exit(EXIT_FAILURE);
    }
}

using the command:

 g++ -o min -I/usr/local/cuda/include  min.cpp -L/usr/local/cuda/lib64  -lcudart

I get:

cudaGetDeviceCount returned 35: CUDA driver version is insufficient for CUDA runtime version

How is this possible, and what can I do about it?

Answer 1

The nVIDIA driver is mis-installed somehow

As @RobertCrovella suggests, it seems I had some apt packages still installed relating to an older driver version - despite what nvidia-smi was telling me. Specifically, nvidia-kernel-dkms was an older version.

Look for the version numbers on the packages you get with sudo dpkg -l "nvidia-*" (and which are installed).

I was not, however, able to figure out exactly what I needed to add/update in order to get my simple program to run. Eventually I caved and went for a manual driver installation.