[英]MXNET - How to add dropout layer to ResNet_v1 pretrained model
I am trying to finetune a pretrained model in mxnet: ResNet50_v1.我正在尝试微调 mxnet 中的预训练模型:ResNet50_v1。 This model does not have dropout and I would like to add it to avoid overfitting and make it look similar to the last layers of I3D_Resnet50_v1_Kinetics400.
该模型没有 dropout,我想添加它以避免过度拟合,并使其看起来类似于 I3D_Resnet50_v1_Kinetics400 的最后一层。 I tried to do the following but when training I get an error:
我尝试执行以下操作,但在训练时出现错误:
Last layers of original network (ResNet50_v1):原始网络的最后几层(ResNet50_v1):
...
(8): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
)
(output): Dense(2048 -> 1000, linear)
My attempt:我的尝试:
classes = 2
model_name = 'ResNet50_v1'
finetune_net = get_model(model_name, pretrained=True)
with finetune_net.name_scope():
finetune_net.output = nn.Dense(2048, in_units=2048)
finetune_net.head = nn.HybridSequential()
finetune_net.head.add(nn.Dropout(0.95))
finetune_net.head.add(nn.Dense(2, in_units=2048))
finetune_net.fc = nn.Dense(2, in_units=2048)
finetune_net.output.initialize(init.Xavier(), ctx = ctx)
finetune_net.head.initialize(init.Xavier(), ctx = ctx)
finetune_net.fc.initialize(init.Xavier(), ctx = ctx)
finetune_net.collect_params().reset_ctx(ctx)
finetune_net.hybridize()
Last layers of the modified network (ResNet50_v1):修改后网络的最后一层(ResNet50_v1):
...
(8): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
)
(output): Dense(2048 -> 2048, linear)
(head): HybridSequential(
(0): Dropout(p = 0.95, axes=())
(1): Dense(2048 -> 2, linear)
)
(fc): Dense(2048 -> 2, linear)
)
Last layers of I3D_Resnet50_v1_Kinetics400: I3D_Resnet50_v1_Kinetics400 的最后几层:
...## Heading ##
(st_avg): GlobalAvgPool3D(size=(1, 1, 1), stride=(1, 1, 1), padding=(0, 0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCDHW)
(head): HybridSequential(
(0): Dropout(p = 0.8, axes=())
(1): Dense(2048 -> 2, linear)
)
(fc): Dense(2048 -> 2, linear)
This is what params of the modifies network look like这就是修改网络的参数的样子
Parameter resnetv10_dense1_weight (shape=(2048, 2048), dtype=float32) write
Parameter resnetv10_dense1_bias (shape=(2048,), dtype=float32) write
Parameter resnetv10_dense2_weight (shape=(2, 2048), dtype=float32) write
Parameter resnetv10_dense2_bias (shape=(2,), dtype=float32) write
Parameter resnetv10_dense3_weight (shape=(2, 2048), dtype=float32) write
Parameter resnetv10_dense3_bias (shape=(2,), dtype=float32) write
Error when training :训练时出错:
/usr/local/lib/python3.7/dist-packages/mxnet/gluon/block.py:825: UserWarning: Parameter resnetv10_dense3_bias, resnetv10_dense3_weight, resnetv10_dense2_bias, resnetv10_dense2_weight is not used by any computation. /usr/local/lib/python3.7/dist-packages/mxnet/gluon/block.py:825:UserWarning:参数 resnetv10_dense3_bias、resnetv10_dense3_weight、resnetv10_dense2_bias、resnetv10_dense2_weight 未被任何计算使用。 Is this intended?
这是故意的吗? out = self.forward(*args)
out = self.forward(*args)
UserWarning: Gradient of Parameter resnetv10_dense2_bias
on context gpu(0) has not been updated by backward since last step
.用户警告:自上
step
以来,上下文 gpu(0) 上的参数resnetv10_dense2_bias
的梯度尚未向后更新。 This could mean a bug in your model that made it only use a subset of the Parameters (Blocks) for this iteration.这可能意味着您的模型中存在一个错误,导致该迭代仅使用参数(块)的一个子集。 If you are intentionally only using a subset, call step with ignore_stale_grad=True to suppress this warning and skip updating of Parameters with stale gradient
如果您有意只使用一个子集,请使用 ignore_stale_grad=True 调用 step 以抑制此警告并跳过更新具有陈旧梯度的参数
dense2 and dense3, the ones I have added as new dense layers are not being updated. dense2 和dense3,我作为新的密集层添加的那些没有被更新。 dense1 was already in the model, I just changed the output from 1000 to 2048.
dense1 已经在模型中,我只是将输出从 1000 更改为 2048。
Any help woul be very much appreciated as I am quite stuck ...任何帮助将不胜感激,因为我很困惑......
Since you assign new layers to the model, you should reimplement hybrid_forward
(or forward
) method to include them in computations:由于您为模型分配了新层,因此您应该重新实现
hybrid_forward
(或forward
)方法以将它们包含在计算中:
from mxnet.gluon import nn
from mxnet.init import Xavier
from mxnet.gluon.block import HybridBlock
class MyResNet(HybridBlock):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.finetune_net = get_model('ResNet50_v1', pretrained=True)
self.finetune_net.output = nn.Dense(2048, in_units=2048)
self.head = nn.HybridSequential()
self.head.add(nn.Dropout(0.95))
self.head.add(nn.Dense(2, in_units=2048))
self.fc = nn.Dense(2, in_units=2048)
def hybrid_forward(self, F, x):
x = self.finetune_net(x)
x = self.head(x)
x = self.fc(x)
return x
def initialize_outputs(self):
self.finetune_net.output.initialize(init=Xavier())
self.head.initialize(init=Xavier())
self.fc.initialize(init=Xavier())
my_resnet = MyResNet()
my_resnet.initialize_outputs()
my_resnet(x)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.