Support for Nvidia CUDA Toolkit 9.2

Question

What is the reasoning that Tensorflow-gpu is bound to a specific version of Nvidia's CUDA Toolkit? The current version appears to look for 9.0 specifically and will not work with anything greater. For example I installed the latest Toolkit 9.2 and added it to path but Tensorflow-gpu will not work with it and complains that it is looking for 9.0.

I can see major version updates not being supported but a minor release?

Answer 1

That's a good question. According to NVidia's website ,

The CUDA driver is backward compatible, meaning that applications compiled against a particular version of the CUDA will continue to work on subsequent (later) driver releases.

So technically, it should not be a problem to support later iterations of a CUDA driver. And in practice, you will find working non-official pre-built binaries with later versions of CUDA and CuDNN on the net [1] , [2] . Even easier to install, the tensorflow-gpu package installed from conda currently comes bundled with CUDA 9.2.

When asked on the topic, a dev answered ,

The answer to why is driver issues in the ones required by 9.1, not many new features we need in cuda 9.1, and a few more minor issues.

So the reason looks rather vague -- he might mean that CUDA 9.1 (and 9.2) requires graphics card driver that are perhaps a bit too recent to be really convenient, but that is an uneducated guess.

If NVidia is right about binary compatibility, you may try to simply rename or link your CUDA 9.2 library as a CUDA 9.0 library and it should work. But I would save all my work before attempting this... and the fact that people go as far as recompiling tensorflow to support later CUDA versions may be a hint on how this could end.

Answer 2

When you download TF, you download a pre-built binary file. In the build process TF is hard linked into a specific version of Cuda, so you cannot use it with different cuda versions.

If you want to work with the new (or sometimes older) version of cuda you will need to install TF from source ( check how here ) Or, if you realy don't want to build yourself, check in these repos, there are others that publish specific TF binaries, few examples:

• https://github.com/mind/wheels
• https://github.com/yaroslavvb/tensorflow-community-wheels
• https://github.com/fo40225/tensorflow-windows-wheel

For your convenience I add here the CUDA + cuDNN versions that are required for each prebuilt Tensorflow version:

(I write here just about the TF versions that I worked with, maybe older TF versions use older versions of CUDA as well)

• before TF v1.5 cuda 8.0 and cuDNN 6
• start from: 1.5 - Prebuilt binaries are now built against CUDA 9 and cuDNN 7.

Answer 3

The issue is not with NVIDIA drivers but Tensorflow itself. I spent an hour trying to make it work, and finally realized that if you download the pre-built binary from googleapi.com, it is hard coded to load libcudart.so.9.0! If you have both cuda 9.0 and 9.2 installed, tensorflow will work (but it's actually loading the dynamic libraries from 9.0). (BTW, I installed TF using anaconda.)

A cleaner approach is to build TF from source. It's not too complicated.