使用 RLlib 时，如何防止我在评估运行期间收到的奖励金额重复出现？

Question

I am using Ray 1.3.0 ( for RLlib ) with a combination of SUMO version 1.9.2 for the simulation of a multi-agent scenario. I have configured RLlib to use a single PPO.network that is commonly updated/used by all N agents. My evaluation settings look like this:

# === Evaluation Settings ===
# Evaluate with every `evaluation_interval` training iterations.
# The evaluation stats will be reported under the "evaluation" metric key.
# Note that evaluation is currently not parallelized, and that for Ape-X
# metrics are already only reported for the lowest epsilon workers.

"evaluation_interval": 20,

# Number of episodes to run per evaluation period. If using multiple
# evaluation workers, we will run at least this many episodes total.

"evaluation_num_episodes": 10,

# Whether to run evaluation in parallel to a Trainer.train() call
# using threading. Default=False.
# E.g. evaluation_interval=2 -> For every other training iteration,
# the Trainer.train() and Trainer.evaluate() calls run in parallel.
# Note: This is experimental. Possible pitfalls could be race conditions
# for weight synching at the beginning of the evaluation loop.

"evaluation_parallel_to_training": False,

# Internal flag that is set to True for evaluation workers.

"in_evaluation": True,

# Typical usage is to pass extra args to evaluation env creator
# and to disable exploration by computing deterministic actions.
# IMPORTANT NOTE: Policy gradient algorithms are able to find the optimal
# policy, even if this is a stochastic one. Setting "explore=False" here
# will result in the evaluation workers not using this optimal policy!

"evaluation_config": {
    # Example: overriding env_config, exploration, etc:
    "lr": 0, # To prevent any kind of learning during evaluation
    "explore": True # As required by PPO (read IMPORTANT NOTE above)
},

# Number of parallel workers to use for evaluation. Note that this is set
# to zero by default, which means evaluation will be run in the trainer
# process (only if evaluation_interval is not None). If you increase this,
# it will increase the Ray resource usage of the trainer since evaluation
# workers are created separately from rollout workers (used to sample data
# for training).

"evaluation_num_workers": 1,

# Customize the evaluation method. This must be a function of signature
# (trainer: Trainer, eval_workers: WorkerSet) -> metrics: dict. See the
# Trainer.evaluate() method to see the default implementation. The
# trainer guarantees all eval workers have the latest policy state before
# this function is called.

"custom_eval_function": None,

What happens is every 20 iterations (each iteration collecting "X" training samples), there is an evaluation run of a minimum of 10 episodes . The sum of reward received by all N agents is summed over these episodes and that is set as the reward sum for that particular evaluation run. Over time, I notice that there is a pattern with the reward sums that repeats over the same interval of evaluation runs continuously, and the learning goes nowhere.

UPDATE (23/06/2021)

Unfortunately, I did not have TensorBoard activated for that particular run but from the mean rewards that were collected during evaluations (that happens every 20 iterations) of 10 episodes each, it is clear that there is a repeating pattern as shown in the annotated plot below:

The 20 agents in the scenario should be learning to avoid colliding but instead continue to somehow stagnate at a certain policy and end up showing the exact same reward sequence during evaluation?

Is this a characteristic of how I have configured the evaluation aspect, or should I be checking something else? I would be grateful if anyone could advise or point me in the right direction.

Thank you.

Answer 1

Step 1 : I noticed that when I stopped the run at some point for any reason, and then restarted it from the saved checkpoint after restoration , most graphs on TensorBoard (including rewards) charted out the line in EXACTLY the same fashion all over again, which made it look like the sequence was repeating.

Step 2 : This led me to believe that there was something wrong with my checkpoints. I compared the weights in checkpoints using a loop and voila, they are all the same! Not a single change! So either there was something wrong with the saving/restoring of checkpoints which after a bit of playing around I found was not the case. So it just meant my weights were not being updated !

Step 3 : I sifted through my training configuration to see if something there was preventing the.network from learning, and I noticed I had set my "multiagent" configuration option "policies_to_train" to a policy that did not exist . This unfortunately, either did not throw a warning/error or it did and I completely missed it.

Solution step : By setting the multiagent "policies_to_train" configuration option correctly, it started to work!

Answer 2

Could it be that due to the multi-agent dynamics, your policy is chasing its tail? How many policies do you have? Are they competing/collaborating/neutral to each other? Note that multi-agent training can be very unstable and seeing these fluctuations is quite normal as the different policies get updated and then have to face different "env"-dynamics b/c of that (env=env+all other policies, which appear as part of the env as well).