pytorch中卷积中的屏蔽内核

Question

Let's say:

x = torch.arange(16, dtype=torch.float).reshape(1, 1, 4, 4)

and a 2d convolution layer is:

layer = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=2, stride=2)
layer.weight.data[:] = 1.
layer.bias.data[:] = 0.

Normally, passing x to layer will give:

>>layer(x)    
tensor([[[[10., 18.],
              [42., 50.]]]], grad_fn=<MkldnnConvolutionBackward>)

Considering having 4 mask filters, how it is done to mask kernel in each step? for example the following picture indicates 4 filters(white: True, black: False)

The output should be:

tensor([[[[5., 15.],
          [30., 40.]]]], grad_fn=<MkldnnConvolutionBackward>)

PS: all masks are obtained by missing pixels in 2d input array. So 4 masks above are actually a matrix with the same shape as input.

Answer 1

I think you are looking for partial convolution from Nvidia research.

A more detailed description is given in their ECCV 2018 paper Image Inpainting for Irregular Holes Using Partial Convolutions

Answer 2

I found the solution.

Element-wise multiplication between input and the mask before feeding it to a Conv2d method would be enough.(masking input is much easier than masking kernel itself !!):

mask = torch.tensor([[[1, 1, 1, 0]], [[1, 0, 1, 1]], [[1, 1, 0, 1]], [[0, 1, 1, 1]]], dtype=torch.float, requires_grad=False).reshape(1, 1, 4, 4)

>>layer(torch.mul(x, mask))
tensor([[[[5., 15.],
          [30., 40.]]]], grad_fn=<MkldnnConvolutionBackward>)

PS Thanks to @Shai I got the idea from partial convolution represented in this paper . However it does some extra manipulation on output. it defines a mask ratio and I guess does some weighting the final output based on it.