增强学习以获取连续的状态和动作空间

Question

Problem

My goal is to apply Reinforcement Learning to predict the next state of an object under a known force in a 3D environment (the approach would be reduced to supervised learning, off-line learning).

Details of my approach

The current state is the vector representing the position of the object in the environment (3 dimensions), and the velocity of the object (3 dimensions). The starting position is randomly initialized in the environment, as well as the starting velocity.

The action is the vector representing the movement from state t to state t+1 .

The reward is just the Euclidean distance between the predicted next state, and the real next state (I already have the target position).

What have I done so far?

I have been looking for many methods to do this. Deep Deterministic Policy Gradients works for a continuous action space, but in my case I also have a continuous state space. If you are interested in this approach, here's the original paper written at DeepMind: http://proceedings.mlr.press/v32/silver14.pdf

The Actor-Critic approach should work, but it is usually (or always) applied to discrete and low-dimensional state space.

Q-Learning and Deep-Q Learning cannot handle high dimensional state space, so my configuration would not work even if discretizing the state space.

Inverse Reinforcement Learning (an instance of Imitation learning, with Behavioral Cloning and Direct Policy Learning ) approximates a reward function when finding the reward function is more complicated than finding the policy function. Interesting approach, but I haven't seen any implementation, and in my case the reward function is pretty straightforward. Is there a methodology to deal with my configuration that I haven't explored?

Answer 1

In your question, I believe there may be a lot of confusion and misconceptions.

1. Firstly, deep deterministic policy gradient (DDPG) can definitely handle continuous states and actions. And it is so famous only because of it. Also, it is the first ever stable architecture to do so. Also, the paper you linked is actually DPG, not DDPG. However, DDPG and DPG can both handle continuous states and actions, but the latter is much more unstable. The paper is actually published by my "senior" at UofA. Heres the link to DDPG: https://arxiv.org/pdf/1509.02971.pdf .

2. Actor-critic RL is not an algorithm, rather, its a family of RL algorithms where the actor maps states to actions, while the critic "pre-processes" the feedback signal so the actor can learn it more efficiently. DDPG is an example of an actor-critic set-up. In DDPG, a DQN is used as a critic to pre-process feedback signals to the deterministic policy gradient (actor).

3. Q-learning and deep Q-learning are also family of RL algorithms. Q-learning certainly cannot handle high state spaces given inadequate computing power, however, deep Q-learning certainly can. An example is Deep Q-network.

Back to the original question.

I can almost guarantee that you can solve your problem using DDPG. In fact, DDPG is still one of the only algorithms that can be used to control an agent in a continuous state, continuous action space.

The other method that can do so is called trust region policy optimization (TRPO). It is developed by the UC Bekelery team (along with OpenAI?). The fundamental structure of TRPO and DDPG are identical (both actor-critic), however, the training is different. DDPG uses a target network approach to guarantee convergence and stability while TRPO puts a Kullerback-Leibler divergence constraint on the update of the networks to ensure each update of the network is not too large (ie optimal policy of the network at t is not too different from t - 1). TRPO is extremely difficult to code, thus, OpenAI published another paper called Proximal Policy Gradient (PPO). This method is similar to TRPO but easier to implement.

Long story short, I'd recommend trying DDPG because if your task is simple as you say, DDPG will definitely work.

Answer 2

Seems like this can be a good paper to look at. If action space is not discretized then it's difficult to specify or select an action from the huge continuous state space. When the action space is discretized, it could lead to a significant loss of information. Presented algorithm starts from actions initialized by the policy network in the discretized space. Then it explores and evaluates actions with the value network using an Upper Confidence Bound (UCB) in the continuous space.

There are couple of other papers might be worth looking at however above paper is most recent one. Hopefully this helps.