Why GPU is much slower than cpu in google colab?

Question

I'm training a RNN on google colab and this is my first time using gpu to train a neural network. From my point of view, GPU should be much faster than cpu, and changing device from cpu to gpu only need to add .to('cuda')<\/code> in the definition of model\/loss\/variable and set google colab 'running on gpu'.

def train(num_epoch = 30,len_vocab = 1, num_hidden=256,embedding_dim = 8,batch_size = 100):
    data = get_data()

    model = MyRNN(len_vocab,num_hidden,embedding_dim).to('cuda') #here 
    if os.path.exists('QingBinLi'):
        model.load_state_dict(torch.load('QingBinLi'))

    criterion = nn.MSELoss().to('cuda')   #here 
    optimizer = torch.optim.Adam(model.parameters(), lr=0.1, weight_decay=1e-5)
    loss_for_draw = []
    model.train()
    data = data.detach().to('cuda') #here 

    for epoch in range(num_epoch+1):

        h = torch.randn(1,batch_size,num_hidden).to('cuda')  #here 
        loss_average = 0
        for i in tqdm(range(data.shape[-2] -batch_size)):
            optimizer.zero_grad()
            pre,h = model(data[:,:,i:i+batch_size,:].squeeze(0) ,h)
            h = h.detach()
            pre = pre.unsqueeze(0).unsqueeze(0)
            loss = criterion(pre, data[:,:,i+1:i+1+batch_size,:].squeeze(0))
            loss_average += loss.item()
            loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), max_norm=10)
            optimizer.step()

        loss_for_draw.append(loss_average/(data.shape[-2] -batch_size))
        torch.save(model.state_dict(), 'QingBinLi')
        print(f'now epoch:{epoch}, loss = {loss_for_draw[-1]}')


    return loss_for_draw

Answer 1

My brother says that when the tensor is very big, such as 1 million dimension, gpu can be faster than cpu, otherwise we don't even need parallel computing because computing are not mainly on tensor multiply, but on copy tensors and other things like that.

So gpu is much slower.

This time gpu is much faster.

So in this kind of computing, gpu is much faster.