OpenAI Gym Atari游戏，TD策略应用

Question

Can I apply TD policy to such environments? Or only methods like DQN and why?

I try to apply TD policy evaluation to Gym's Atari games' simulations in Python and I am a little new to it. I have this Value class:

class V_Class():  
""" Class to store the state Value function
    V(s) = expected future discounted reward from s onwards (the return Gt)
    Stores it as a dictionnary and adds states as encounteded (get method)
    Two methods: get and set
"""
def __init__(self):       
    self.f = {}

def get(self, s):        
    if(s not in self.f):
        self.f[s] = 0             
    return self.f[s]

def set(self, s, y):
    self.f[s] = y  

and I have this implementation:

env = Environment.Environment("SpaceInvaders-v0")
V = V_Class()

iepisode = 0
while iepisode <= 1:
    obs = env.reset()
    done = False
    SUMREWARD=0
    while not done:
        print("obs:", obs)
        action = env.action_space.sample()
        new_obs, reward, done, info = env.step(action)
        SUMREWARD+=reward
        new_Vs = V.get(obs) + 0.7*(reward + 0.5*V.get(new_obs) - V.get(obs)) 
        V.set(obs,new_Vs)
        obs = new_obs

but I get this error:

TypeError: unhashable type: 'numpy.ndarray'  as you can see here:

>  TypeError             Traceback (most recent call
> last) <ipython-input-12-428939358367> in <module>
>      12         new_obs, reward, done, info = env.step(action)
>      13         SUMREWARD+=reward
> ---> 14         new_Vs = V.get(obs) + 0.7*(reward + 0.5*V.get(new_obs) - V.get(obs))
>      15         V.set(obs,new_Vs)
>      16         obs = new_obs
> 
> <ipython-input-4-5d3d077cd162> in get(self, s)
>       9 
>      10     def get(self, s):
> ---> 11         if(s not in self.f):
>      12             self.f[s] = 0
>      13         return self.f[s]
> 
> TypeError: unhashable type: 'numpy.ndarray'

Because this V class was initially made for classical environments where the states (obs variable) are single numbers, while Atari environments have big, 3 - dimensional numpy.ndarray , representing the states.

V class here should check in its dictionary f , if this state has an already stored value, and if not to store a value for it based on the formula:

new_Vs = V.get(obs) + 0.7*(reward + 0.5*V.get(new_obs) - V.get(obs))

How would you suggest me to fix this? Is there a process which I don't know an I should follow for such cases or I just have to update my V class methods to store big states as a dictionary key?

Answer 1

You are trying to search in a dictionary( f ) with a numpy array ( obs ) as the key like in this example:

import numpy as np
array = np.ndarray([1,2,3])
dict = {}
if array not in dict:
    print("Its not")
else:
    print("Its in")

that returns the same error:

TypeError: unhashable type: 'numpy.ndarray'

You have to use as a key a hashable type, not an array.