超参数调整与射线不停止

Question

I recently tried to use a hyperOpt algorithm to find the best hyperparameters configuration of a PPO algorithm.

The trained model is en Gym environment (LunarLander-v2).

When I run my program, it never stops, I did not find which parameter add to the config to limit the number of episodes of my training.

Here is the config I used: ''' def explore(config): # ensure we collect enough timesteps to do sgd if config["train_batch_size"] < config["sgd_minibatch_size"] * 2: config["train_batch_size"] = config["sgd_minibatch_size"] * 2 # ensure we run at least one sgd iter if config["num_sgd_iter"] < 1: config["num_sgd_iter"] = 1 return config

    config = {
                 "env": "LunarLander-v2",
                 "sgd_minibatch_size": 5000,
                 "num_sgd_iter": 2,
                 "lr": tune.uniform(5e-6, 5e-2),
                 "lambda": tune.uniform(0.6, 0.9),
                 "vf_loss_coeff": 0.7,
                 "kl_target": 0.01,
                 "kl_coeff": tune.uniform(0.5, 0.9),
                 "entropy_coeff": 0.001,
                 "clip_param": tune.uniform(0.4, 0.8),
                 "train_batch_size": 25000, # taille de l'épisode
                 # "monitor": True,
                 # "model": {"free_log_std": True},
                 "num_workers": 4,
                 "num_gpus": 0,
                 # "rollout_fragment_length":3
                 # "batch_mode": "complete_episodes"
             }

    optimizer = HyperOptSearch(metric="episode_reward_mean", mode="max", n_initial_points=1, random_state_seed=7, space=explore(config))

    # optimizer = ConcurrencyLimiter(optimizer, max_concurrent=4)

    analysis = tune.Tuner(
        "PPO",  # Objective function

        tune_config=tune.TuneConfig(

        metric="episode_reward_mean", # the metric we want to study
        mode="max", # maximize the metric
        search_alg=optimizer,
        # num_samples will repeat the entire config 'num_samples' times == Number of trials dans l'output 'Status'
        num_samples=2,

        ),

    )
    results = analysis.fit()'''

I thought the hyperparameter for the number of episodes was num_sgd_iter, but visibly it is not.

Answer 1

tune.Tuner has a run_config argument, which you can use as follows in your case:

 analysis = tune.Tuner(
        "PPO",  # Objective function
        tune_config=tune.TuneConfig(
            search_alg=optimizer,
            num_samples=2,
        ),
        run_config=air.RunConfig(
            stop={"num_env_steps_trained": 10000}
        ),
    )

You can generally stop on anything that tune reports back (things that you see it report in your CLI). Be aware that some of these are nested and so you might have to specify them according to their nesting. Also, metric="episode_reward_mean" and mode="max" are the defaults, so you don't have to specify them.