使用 Pytorch model 参数进行计算时出现结果类型转换错误

Question

When I ran the code below:

import torchvision

model = torchvision.models.densenet201(num_classes=10)
params = model.state_dict()
for var in params:
    params[var] *= 0.1

a RuntimeError was reported:

RuntimeError: result type Float can't be cast to the desired output type Long

But when I changed params[var] *= 0.1 to params[var] = params[var] * 0.1 , the error disappears.

Why would this happen?

I thought params[var] *= 0.1 had the same effect as params[var] = params[var] * 0.1 .

Answer 1

First, let us know the first long-type parameter in dens.net201 , you will find the features.norm0.num_batches_tracked which indicates the number of mini-batches during training used to calculate the mean and variance if there is BatchNormalization layer in the model. This parameter is a long-type number and cannot be float type because it behaves like a counter .

Second, in PyTorch, there are two types of operations:

• Non-Inplace operations: you assign the new output after calculation to a new copy from the variable, eg x = x + 1 or x = x / 2. The memory location of x before assignment not equal to the memory location after assignment because you have a copy from the original variable.
• Inplace operations: when the calculations directly applied to the original copy of the variable without making any copy here eg x += 1 or x /= 2.

Let's move to your example to understand what happened:

1. Non-Inplcae operation:

    model = torchvision.models.dens.net201(num_classes=10) params = model.state_dict() name = 'features.norm0.num_batches_tracked' print(id(params[name])) # 140247785908560 params[name] = params[name] + 0.1 print(id(params[name])) # 140247785908368 print(params[name].type()) # changed to torch.FloatTensor

2. Inplace operation:

    print(id(params[name])) # 140247785908560 params[name] += 1 print(id(params[name])) # 140247785908560 print(params[name].type()) # still torch.LongTensor params[name] += 0.1 # you want to change the original copy type to float,you got an error

Finally, some remarks:

• In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss of history. Hence, their use is discouraged. Source
• You should be cautious when you decide to use in-place operations since they overwrite the original content.
• If you use pandas, this is a bit similar to the inplace=True in pandas:).

This is a good resource to read more about in-place operation source and read also this discussion source .