bazel build tensorflow on windows 10 getting cudnn.h- system cannot find the file specified

Question

I keep getting this error when trying to build the tensorflow-gpu using bazel and python in Windows 10:

Cuda Configuration Error: Error reading C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2/cuda/include/cudnn.h: ja
va.io.IOException: ERROR: src/main/native/windows/processes-jni.cc(239): CreateProcessW("grep" --color=never -A1 -E "#de
fine CUDNN_MAJOR" "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2/cuda/include/cudnn.h"): The system cannot fin
d the file specified.

This is the command I'm trying to run:

bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package

I've confirmed the C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2/cuda/include/cudnn.h is there.

I've tried running it from VS2015 x64 Native Tools Command Prompt, cmd, and powershell and get the same error.

I'm using bazel 0.16.1, CUDA 9.2, Anaconda3 (Python 3.6.5), and CUDNN 7.2.1. I "installed" the CUDDNN files by unzipping its cuda folder into my C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v9.2 folder (ie the whole "cuda" folder). I specified the C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v9.2\\cuda path in the cudnn path question when I ran the configure.py. The configure.py completes without error.

I also tried putting the CUDNN files directly in the C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v9.2 folder (instead of a cuda folder in there) and specified the default location and still get basically the same error: "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2/include/cudnn.h": The system cannot find the file specified.

I've added these to my path (and rebooted):

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\libnvvp
C:\tools\msys64
C:\tools\bazel
C:\tools\bazel\bazel.exe
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\extras\CUPTI\libx64

These are the system variables I've set:

BAZEL_SH  C:\tools\msys64\usr\bin\bash.exe
BAZEL_VC  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC
BAZEL_VS  C:\Program Files (x86)\Microsoft Visual Studio 14.0

This is the full error:

ERROR: Skipping '//tensorflow/tools/pip_package:build_pip_package': error loading package 'tensorflow/tools/pip_package'
: Encountered error while reading extension file 'cuda/build_defs.bzl': no such package '@local_config_cuda//cuda': Trac
eback (most recent call last):
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 1458
                _create_local_cuda_repository(repository_ctx)
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 1185, in _create_local_cuda_repository
                _get_cuda_config(repository_ctx)
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 909, in _get_cuda_config
                _cudnn_version(repository_ctx, cudnn_install_base..., ...)
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 584, in _cudnn_version
                find_cuda_define(repository_ctx, cudnn_header_dir, "c...", ...)
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 534, in find_cuda_define
                auto_configure_fail(("Error reading %s: %s" % (str(h...)))
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 315, in auto_configure_fail
                fail(("\n%sCuda Configuration Error:%...)))

Cuda Configuration Error: Error reading C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2/cuda/include/cudnn.h: ja
va.io.IOException: ERROR: src/main/native/windows/processes-jni.cc(239): CreateProcessW("grep" --color=never -A1 -E "#de
fine CUDNN_MAJOR" "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2/cuda/include/cudnn.h"): The system cannot fin
d the file specified.

WARNING: Target pattern parsing failed.
ERROR: error loading package 'tensorflow/tools/pip_package': Encountered error while reading extension file 'cuda/build_
defs.bzl': no such package '@local_config_cuda//cuda': Traceback (most recent call last):
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 1458
                _create_local_cuda_repository(repository_ctx)
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 1185, in _create_local_cuda_repository
                _get_cuda_config(repository_ctx)
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 909, in _get_cuda_config
                _cudnn_version(repository_ctx, cudnn_install_base..., ...)
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 584, in _cudnn_version
                find_cuda_define(repository_ctx, cudnn_header_dir, "c...", ...)
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 534, in find_cuda_define
                auto_configure_fail(("Error reading %s: %s" % (str(h...)))
        File "C:/tensorflow/third_party/gpus/cuda_configure.bzl", line 315, in auto_configure_fail
                fail(("\n%sCuda Configuration Error:%...)))

Cuda Configuration Error: Error reading C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2/cuda/include/cudnn.h: ja
va.io.IOException: ERROR: src/main/native/windows/processes-jni.cc(239): CreateProcessW("grep" --color=never -A1 -E "#de
fine CUDNN_MAJOR" "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2/cuda/include/cudnn.h"): The system cannot fin
d the file specified.

Any ideas what I could have done wrong, or what I might have missed, or what I could check next?

Answer 1

The problem is a bug in cuda_configure.bzl : it uses ctx.execute instead of ctx.action.run_shell . If you'd be so kind to file a bug to TensorFlow , that'd be great!

To work around it: add c:\\tools\\msys64\\usr\\bin to your PATH. That's where grep.exe is so ctx.execute can find grep on the PATH.