How to represent state space when designing a grid world for reinforcement learning

Question

I want to design a say 5x5 grid world in which an agent can move to experiment a bit with RL algorithms. Intuitively, I would describe the states by tuples (x,y), ie in python by using lists [x,y] or numpy arrays. However, this becomes a nuisance when implementing most algorithms. For instance, if I want a Q-value matrix with entries Q(s,a), I can't just use a numpy matrix, where the row index corresponds to the state, but must use something more complicated.

My question is whether it is standard to just enumerate all states, ie 1, 2, ..., 25 instead of using (x,y), or whether there is another clever way to represent states that makes handling them easy as well.

Answer 1

You could make use of a conversion function for state index to coordinates on the grid. Then you can keep track of the state indices in your Q-table and use the conversion function to refer to the grid cell. Something like this might work for you (coordinate to index):

idx = point[0] + point[1] * grid_size

Hope this helps!