Pytorch 中 Sigmoid 函数破坏的梯度计算

Question

Hey I have been struggling with this weird problem. Here is my code for the Neural Net:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv_3d_=nn.Sequential(
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU(),
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU(),
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU()  
        )

        self.linear_layers_ = nn.Sequential(

            nn.Linear(batch_size*32*32*32,batch_size*32*32*3),
            nn.LeakyReLU(),
            nn.Linear(batch_size*32*32*3,batch_size*32*32*3),
            nn.Sigmoid()
        )

    def forward(self,x,y,z):
        conv_layer = x + y + z
        conv_layer = self.conv_3d_(conv_layer)
        conv_layer = torch.flatten(conv_layer)
        conv_layer = self.linear_layers_(conv_layer)
        conv_layer = conv_layer.view((batch_size,3,input_sizes,input_sizes))
        return conv_layer

The weird problem I am facing is that running this NN gives me an error

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3072]], which is output 0 of SigmoidBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The stack trace shows that the issue is in line

conv_layer = self.linear_layers_(conv_layer)

However, if I replace the last activation function of my FCN from nn.Sigmoid() to nn.LeakyRelu(), the NN executes properly.

Can anyone tell me why Sigmoid activation function is causing my backward computation to break?

Answer 1

I found the problem with my code. I delved deeper into what in-place actually meant. So, if you check the line

conv_layer = self.linear_layers_(conv_layer)

linear_layers_ of the assignment is changing the values of conv_layer in-place and as a result the values are getting overwritten and because of this, gradient computation fails. Easy solution for this problem is to use the clone() function

ie

conv_layer = self.linear_layers_(conv_layer).clone()

This creates a copy of the right hand computation and Autograd is able to store the reference of the computation graph.