Pytorch：权重未更新

Question

I'm trying to train a model, and it doesn't work because weights aren't updating when I call the following:

self.optimizer = Adam(self.PPO.parameters(), lr=0.1, eps=epsilon)
total_loss = Variable(policy_loss + 0.5*value_loss - entropy_loss.mean() * 0.01, requires_grad=True)

self.optimizer.zero_grad()
(total_loss * 10).backward()
self.optimizer.step()

When I print the weights, they're all the same (loss isn't zero, and learning rate set to 0.1), and when I compare them (even with clone() called on each param) it always returns True. Total loss has a grad_fn attribute too... The optimizer is created in the constructor of my agent class.

My code is based on this repository: https://github.com/andreiliphd/tennis-ppo/blob/master/agent.py

This is my agent constructor:

    def __init__(self, PPO, learning_rate, epsilon, discount_rate, entropy_coefficient, ppo_clip, gradient_clip,
                 rollout_length, tau):
        self.PPO = PPO
        self.learning_rate = learning_rate
        self.epsilon = epsilon
        self.discount_rate = discount_rate
        self.entropy_coefficient = entropy_coefficient
        self.ppo_clip = 0.2
        self.gradient_clip = 5
        self.rollout_length = rollout_length
        self.tau = tau
        self.optimizer = Adam(self.PPO.actor.parameters(), lr=0.1, eps=epsilon)
        self.device = torch.device('cpu')

This is my PPO class, which creates two networks with a forward function, and some hidden layers

class PPO(nn.Module):

    def __init__(self, state_shape, action_num, mlp_layers, device=torch.device('cpu')):
        super(PPO, self).__init__()
        self.state_shape = state_shape
        self.action_num = action_num
        self.mlp_layers = mlp_layers
        self.device = torch.device('cpu')

        layer_dims = [np.prod(self.state_shape)] + self.mlp_layers
        self.actor = PPO_Network(state_shape, action_num, layer_dims, True)
        self.actor = self.actor.to(device)
        self.critic = PPO_Network(state_shape, 1, layer_dims, False)
        self.critic = self.critic.to(device)
        self.to(device)

Any indications on why this is happening, and what I am overlooking are very welcome. :) I can give more info or code if needed.

Answer 1

I fixed this:) It was very stupid, it was because I was converted one of the values returned by my networks to numpy, and then converted it back to a tensor. It took me a while to realize because of some messy code.