Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed

Question

I'm doing a project on face recognition on Google colab. When I try to execute the following code

H = model.fit(
    aug.flow(trainX, trainY, batch_size=BS),
    steps_per_epoch=len(trainX) // BS,
    validation_data=(testX, testY),
    validation_steps=len(testX) // BS,
    epochs=EPOCHS)

it gives me this error

Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[node model/Conv1/Conv2D
 (defined at /usr/local/lib/python3.7/dist-packages/keras/layers/convolutional.py:238)
]] [Op:__inference_train_function_7525]

Errors may have originated from an input operation.
Input Source operations connected to node model/Conv1/Conv2D:
In[0] IteratorGetNext (defined at /usr/local/lib/python3.7/dist-packages/keras/engine/training.py:866)  
In[1] model/Conv1/Conv2D/ReadVariableOp:

There is a lot more in the error it goes on.. and I did try restarting the runtime and most solutions to this problem are on local machines. Please help me out if anyone knows the solution

tensorflow version 2.7.0 CUDA Version: 11.2

Answer 1

You can turn on memory growth by calling tf.config.experimental.set_memory_growth , which attempts to allocate only as much GPU memory as needed for the runtime allocations: it starts out allocating very little memory, then as the model trains and more GPU memory is needed, the GPU memory is extended.To turn on memory growth for a specific GPU, use the following code prior to allocating any tensors or executing any ops.

def solve_cudnn_error():
    gpus = tf.config.experimental.list_physical_devices('GPU')
    if gpus:
        try:
            # Currently, memory growth needs to be the same across GPUs
            for gpu in gpus:
                tf.config.experimental.set_memory_growth(gpu, True)
            logical_gpus = tf.config.experimental.list_logical_devices('GPU')
            print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
        except RuntimeError as e:
            # Memory growth must be set before GPUs have been initialized
            print(e)