[英]How to mask weights in PyTorch weight parameters?
I am attempting to mask (force to zero) specific weight values in PyTorch.我试图在 PyTorch 中屏蔽(强制为零)特定的权重值。 The weights I am trying to mask are defined as so in the def __init__
我试图屏蔽的权def __init__
是这样定义的
class LSTM_MASK(nn.Module):
def __init__(self, options, inp_dim):
super(LSTM_MASK, self).__init__()
....
self.wfx = nn.Linear(input_dim, curernt_output, bias=add_bias)
The mask is also defined in def __init__
as掩码也在def __init__
定义为
self.mask_use = torch.Tensor(curernt_output, input_dim)
The mask is a constant and the .requires_grad_()
is False
for the mask parameter.掩码是一个常量,掩码参数的.requires_grad_()
为False
。 Now in the def forward
part of the class I attempt to do an element-wise multiplication of the weight parameter and the mask before the linear operation is completed现在在类的def forward
部分中,我尝试在线性运算完成之前对权重参数和掩码进行元素乘法
def forward(self, x):
....
self.wfx.weight = self.wfx.weight * self.mask_use
wfx_out = self.wfx(x)
I get ann error message:我收到一条错误消息:
self.wfx.weight = self.wfx.weight * self.mask_use
File "/home/xyz/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 537, in __setattr__
.format(torch.typename(value), name))
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
But when I check on the both parameters with .type()
both of them come up as torch.cuda.FloatTensor
.但是当我用.type()
检查这两个参数时,它们都作为torch.cuda.FloatTensor
。 I am not sure why there is an error here.我不确定为什么这里有错误。
The element-wise operation always returns a FloatTensor
.按元素操作总是返回一个FloatTensor
。 It is not possible to assign normal tensors as weight
of layers.不可能将正常张量指定为层的weight
。
There are two possible options to deal with it.有两种可能的选择来处理它。 You can assign it to the data
attribute of your weight, there it is possible assign normal tensors.您可以将其分配给您的权重的data
属性,在那里可以分配正常的张量。
Or alternatively you convert your result to an nn.Parameter
itself, then you can assign it to wfx.weight
.或者,您将结果转换为nn.Parameter
本身,然后您可以将其分配给wfx.weight
。
Here is an example which shows both ways:这是一个显示两种方式的示例:
import torch
import torch.nn as nn
wfx = nn.Linear(10, 10)
mask_use = torch.rand(10, 10)
#wfx.weight = wfx.weight * mask_use #your example - this raises an error
# Option 1: write directly to data
wfx.weight.data = wfx.weight * mask_use
# Option 2: convert result to nn.Parameter and write to weight
wfx.weight = nn.Parameter(wfx.weight * mask_use)
Disclaimer: When using an =
(assignment) on the weights you are replacing the weights tensor of your parameter.免责声明:在权重上使用=
(赋值)时,您将替换参数的权重张量。 This may have unwanted effects on the graph resp.这可能会对图形产生不良影响。 the optimization step.优化步骤。
One effective way change your Tensorfloat variable to parameter variable:将 Tensorfloat 变量更改为参数变量的一种有效方法:
self.wfx.weight = torch.nn.parameter.Parameter((self.wfx.weight.data * self.mask_use))
I hope this would be usefull.我希望这会很有用。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.