在强化学习中将离散动作转换为连续动作

Question

In reinforcement learning, we empirically know using discrete actions is easier to train than using continuous actions.

But theoretically, continuous actions is more accurate and fast, just like our human, most of our actions are continuous.

So is there any method or related research that train a discrete action policy for easier start and then transfer that policy to output continuous actions for better precision?

Thanks.

Answer 1

You can certainly do that, any papers that does continuous control using reinforcement learning will do this. The only ones that don't are the researchers that use deep reinforcement learning or reinforcement learning with function approximation. My research is applying both reinforcement learning and deep reinforcement learning on dynamical systems. I discretize my state and action space to adequate resolution, and then apply it to control problems.

I am currently working on some methods to make the discretized system work for continuous spaces. One method is to use linear interpolation. If your state falls between 2 discretized points, you can use linear interpolation to identify the optimal action (in the continuous space). It works especially well for linear system since the control law is linear as follows:

u = Kx

And this method is directly in line to what you ask: training on a discrete space, and then applying it to a continuous control problem.

However, traditionally, continuous control problems are solved using either linear function approximation such as tile coding, or non-linear function approximation such as artificial neural networks. These methods are more advanced, I would suggest trying to use more basic discrete RL methods first. I have a RL code on my Github you can use, let me know if you have any issues.