Running Tensorflow/Keras Using GPU with CUDA, cuDNN, Anaconda, RTX 3060 Ti

Question

I am attempting to train a neural network using my new RTX 3060 Ti for the first time and have encountered a difficult error. Below is the error message:

2020-12-17 12:45:09.600373: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
  File "marge.py", line 365, in <module>
    MARGE(*sys.argv[1:])
  File "marge.py", line 357, in MARGE
    filters, filt2um)
  File "lib\NN.py", line 712, in driver
    nn.train(train_batches, valid_batches, epochs, patience)
  File "lib\NN.py", line 335, in train
    model_checkpoint])
  File "C:\Users\Nick\anaconda3\envs\marge\lib\site-packages\keras\engine\training.py", line 1039, in fit
    validation_steps=validation_steps)
  File "C:\Users\Nick\anaconda3\envs\marge\lib\site-packages\keras\engine\training_arrays.py", line 154, in fit_loop
    outs = f(ins)
  File "C:\Users\Nick\anaconda3\envs\marge\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "C:\Users\Nick\anaconda3\envs\marge\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Users\Nick\anaconda3\envs\marge\lib\site-packages\tensorflow\python\client\session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Blas GEMM launch failed : a.shape=(256, 7), b.shape=(7, 4096), m=256, n=4096, k=7
         [[{{node dense_1/MatMul}}]]
         [[loss/mul/_125]]
  (1) Internal: Blas GEMM launch failed : a.shape=(256, 7), b.shape=(7, 4096), m=256, n=4096, k=7
         [[{{node dense_1/MatMul}}]]
0 successful operations.
0 derived errors ignored.

I am working within a Python 3.7.2 Anaconda environment on Windows 10. Here are (what I believe to be) the relevant packages installed on this environment:

cudnn                     7.6.5
cudnn                     7.6.5
keras                     2.2.4
keras-applications        1.0.8
keras-base                2.2.4
keras-preprocessing       1.1.0
tensorflow                1.14.0
tensorflow-base           1.14.0
tensorflow-estimator      1.14.0
tensorflow-gpu            1.14.0

Things I have tried:

• Installing the NVidia version of Tensorflow, but the commands pip install --user nvidia-pyindex and pip install --user nvidia-tensorflow[horovod] resulted in errors.

• Adding the following to the top of my code:

  config = tf.ConfigProto()

  config.gpu_options.allow_growth = True

  session = tf.Session(config=config)

This didn't work either ("failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED"), but it seems that my GPU was recognized:

2020-12-17 12:27:24.445007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 3060 Ti major: 8 minor: 6 memoryClockRate(GHz): 1.71
pciBusID: 0000:06:00.0
2020-12-17 12:27:24.445413: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-12-17 12:27:24.445639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-12-17 12:27:24.445772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-17 12:27:24.445881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2020-12-17 12:27:24.445971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2020-12-17 12:27:24.446221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6712 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3060 Ti, pci bus id: 0000:06:00.0, compute capability: 8.6)

Is it possible that the RTX 3060 Ti is not supported for this usage yet?

Please let me know if there is any additional information I can provide. Thanks in advance for your help!

EDIT: I have also tried the suggestions of these tutorials (in retrospect, installing CUDA and cuDNN seems inmportant). I also ran the commands:

tf.test.is_built_with_cuda()
tf.test.is_gpu_available(cuda_only=False, min_cuda_compute_capability=None)

both of which returned "True". However, I still received the "failed to run cuBLAS routine" error. I also noticed that the 3060 Ti does not appear on NVIDEA's list of CUDA-compatible GPU's, so maybe I am just out of luck...

Answer 1

I also had some struggles like you did when I was trying to run on my GPU. I don't remember well, so I am afraid that I might not be much of a help.

I see that you are using below the second version of the TensorFlow. You have to install the right Cuda and cuDNN version for the TensorFlow 1.14 you are using.

Here is all the link you should try to install:

• Driver for your GPU:https://www.nvidia.com/Download/driverResults.aspx/167753/en-us
• The right CUDA version: https://developer.nvidia.com/cuda-10.0-download-archive
• The right cuDNN version (you need to create an account but download the 7.4 version): https://developer.nvidia.com/cudnn

I hope this works and let me know if it does, good luck!