Requesting a particular type of GPU on Google Colab

Question

I am using Google Colab to run some deep learning code.

In previous days it would run okay, but recently, without making changes to the code I got the error:

RuntimeError: CUDA call failed (correlation_forward_cuda at correlation_cuda.cc:80)

Apparently this is an error that can appear depending on the graphics card or system that you are using to run the code.

On Google Colab, I just restarted the runtime and tried the code again. This time it worked - so I just suspected that it was down to luck - depending if you were given a GPU that was compatible or not.

The problem is that now EVERY time I run the code I get the error message, no matter how many times I restart the runtime or close my notebook or change runtime type to None and then back to GPU - I am now always getting the error message.

Is there a way that I can request a specific type of GPU? I'm not sure what type yet - but there was certainly one version that ran my code successfully at one point.

My code takes a while to even get to the point of failure, so it's very frustrating to keep running the code over and over hoping that one time it will work.

Answer 1

You can't select a specific GPU. What you can do after connecting to a VM is to check the assigned GPU with the command:

!nvidia-smi --query-gpu=gpu_name,driver_version,memory.total --format=csv

If the given is not what you desired, you can terminate the machine at the "Manage sessions" menu (refresh the page if the button is not working). If you are lucky, by connecting again to another machine, you will obtain the GPU you wanted, else you can repeat the process until then.

I see that the link you posted is related to a DAIN app. I don't know that concrete case or if you are trying to execute it, but I tried the DAIN network on Colab and only worked with P1000 and P4. T4 and K80 failed throwing the same error that you refer to.