When trying to build TensorFlow from source: Inconsistent CUDA toolkit path: /usr vs /usr/lib

Question

On a fresh LambdaLabs GPU instance I install Bazel with Bazelisk:

wget https://github.com/bazelbuild/bazelisk/releases/download/v1.8.1/bazelisk-linux-amd64
chmod +x bazelisk-linux-amd64
sudo mv bazelisk-linux-amd64 /usr/local/bin/bazel

Then I download the TF source:

git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout r2.11

Then I run the config:

./configure

And this is the result:

ubuntu@*********:~/tensorflow$ ./configure
You have bazel 5.3.0 installed.
Please specify the location of python. [Default is /usr/bin/python3]: 


Found possible Python library paths:
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.8/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.

Inconsistent CUDA toolkit path: /usr vs /usr/lib
Asking for detailed CUDA configuration...

Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 11]: 

Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 2]: 

Please specify the TensorRT version you want to use. [Leave empty to default to TensorRT 6]: 

Please specify the locally installed NCCL version you want to use. [Leave empty to use http://github.com/nvidia/nccl]: 

Please specify the comma-separated list of base paths to look for CUDA libraries and headers. [Leave empty to use the default]: 

Inconsistent CUDA toolkit path: /usr vs /usr/lib
Asking for detailed CUDA configuration...

I have no idea how to tell the compiler which CUDA toolkit path to use, and even which is the correct path. I'm just trying to rebuild TF with TFRT support.

Thanks

EDIT:

When I checked where CUDA is installed:

locate cuda | grep /cuda$

/home/ubuntu/tensorflow/tensorflow/compiler/xla/stream_executor/cuda
/home/ubuntu/tensorflow/tensorflow/stream_executor/cuda
/home/ubuntu/tensorflow/third_party/gpus/cuda
/usr/include/cuda
/usr/include/thrust/system/cuda
/usr/lib/cuda
/usr/lib/python3/dist-packages/pycuda/cuda
/usr/lib/python3/dist-packages/tensorflow/include/tensorflow/stream_executor/cuda
/usr/lib/python3/dist-packages/theano/sandbox/cuda
/usr/lib/python3/dist-packages/torch/cuda
/usr/lib/python3/dist-packages/torch/backends/cuda
/usr/lib/python3/dist-packages/torch/include/ATen/cuda
/usr/lib/python3/dist-packages/torch/include/ATen/native/cuda
/usr/lib/python3/dist-packages/torch/include/c10/cuda
/usr/lib/python3/dist-packages/torch/include/torch/csrc/cuda
/usr/lib/python3/dist-packages/torch/include/torch/csrc/jit/cuda
/usr/lib/python3/dist-packages/torch/include/torch/csrc/jit/codegen/cuda
/usr/lib/python3/dist-packages/torch/include/torch/csrc/jit/codegen/fuser/cuda
/usr/share/doc/libthrust-dev/examples/cuda

Apparently the correct path is /usr/lib/ but I don't know how to tell the compiler to use that path.

Answer 1

can you run these commands? If see versions then build with Bazel see any error? My environment is not different I running on Windows 10 no simulation but as warning messages they will not allowed in next versions for GPU support ( actually they warning for many time but I still using Windows 10 as many users )

Matches of requirements and expectations, user and application meet at the level of low-level standards

1. nvdisasm --version ( to see CUDA versions running )

    C:\WINDOWS\system32>nvdisasm --version
    nvdisasm: NVIDIA (R) CUDA disassembler
    Copyright (c) 2005-2021 NVIDIA Corporation
    Built on Sun_Aug_15_21:12:33_Pacific_Daylight_Time_2021
    Cuda compilation tools, release 11.4, V11.4.120
    Build cuda_11.4.r11.4/compiler.30300941_0

2. nvcc -V ( to see CUDA drivers versions running )

    C:\WINDOWS\system32>nvcc -V
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2021 NVIDIA Corporation
    Built on Sun_Aug_15_21:18:57_Pacific_Daylight_Time_2021
    Cuda compilation tools, release 11.4, V11.4.120
    Build cuda_11.4.r11.4/compiler.30300941_0

3. nvidia-smi ( to see the specification and support versions *11.6 )

    C:\WINDOWS\system32>nvidia-smi
    Tue Nov  8 00:34:17 2022
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 512.15       Driver Version: 512.15       CUDA Version: 11.6     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  NVIDIA GeForce ... WDDM  | 00000000:01:00.0  On |                  N/A |
    |  0%   45C    P8     9W / 120W |   1026MiB /  6144MiB |      2%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

4. python./configure.py

    C:\Python310\tensorflow>python ./configure.py
    You have bazel 6.0.0-pre.20221020.1 installed.
    Please specify the location of python. [Default is C:\Python310\python.exe]:

    Found possible Python library paths:
      C:\Python310\lib\site-packages
      \Python310\object_detection\models
    Please input the desired Python library path to use.  Default is [C:\Python310\lib\site-packages]

    Do you wish to build TensorFlow with ROCm support? [y/N]: n
    No ROCm support will be enabled for TensorFlow.
    
    
    WARNING: Cannot build with CUDA support on Windows.
    Starting in TF 2.11, CUDA build is not supported for Windows. For using TensorFlow GPU on Windows, you will need to build/install TensorFlow in WSL2.
    
    Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is /arch:AVX]:
    
    
    Would you like to override eigen strong inline for some C++ compilation to reduce the compilation time? [Y/n]: y
    Eigen strong inline overridden.
    
    Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
    Not configuring the WORKSPACE for Android builds.

5. bazel build //tensorflow/tools/pip_package:build_pip_package