了解 RLLIB 中 PPO 的张量板图

Question

I am beginner in Deep RL and would like to train my own gym environment in RLLIB with the PPO algorithm. However, I am having some difficulties seeing if my hyperparameter settings are being successful. Apart from the obvious episode_reward_mean metric which should rise we have many other plots.

I am especially interested in how entropy should evolve during a successful training. In my case it looks like this:

entropy.jpg

It is usually dropping below 0 and then converging. I understand that entropy as part of the loss function is enforcing exploration and can therefore speedup learning. But why is it getting negative? Shouldn't it be always greater or equal to 0?

What are other characteristics of a successful training (vf_explained_var, vf_loss, kl,...)?

Answer 1

If your action space is continuous, entropy can be negative, because differential entropy can be negative .

Ideally, you want the entropy to be decreasing slowly and smoothly over the course of training, as the agent trades exploration in favor of exploitation.

With regards to the vf_* metrics, it's helpful to know what they mean.

In policy gradient methods, it can be helpful to reduce the variance of rollout estimates by using a value function--parameterized by a neural network--to estimate rewards that are farther in the future (check the PPO paper for some math on page 5).

vf_explained_var is the explained variation of those future rewards through the use of the value function. You want this to be higher if possible, and it tops out at 1; however, if there is randomness in your environment it's unlikely for this to actually hit 1. vf_loss is the error that your value function is incurring; ideally this would decrease to 0, though this isn't always possible (due to randomness). kl is the difference between your old strategy and your new strategy at each time step: you want this to smoothly decrease as you train to indicate convergence.