编码马尔可夫决策过程的问题

Question

I am trying to code Markov-Decision Process (MDP) and I face with some problem. Could you please check my code and find why it isn't works

I have tried to do make it with some small data and it works and give me necessary results, which I feel is correct. But my problem is with generalising of this code. Yeah, I know about MDP library, but I need to code this one. This code works and I want same result in class:

import pandas as pd
data = [['3 0', 'UP', 0.6, '3 1', 5, 'YES'], ['3 0', 'UP', 0.4, '3 2', -10, 'YES'], \
    ['3 0', 'RIGHT', 1, '3 3', 10, 'YES'], ['3 1', 'RIGHT', 1, '3 3', 4, 'NO'], \
    ['3 2', 'DOWN', 0.6, '3 3', 3, 'NO'], ['3 2', 'DOWN', 0.4, '3 1', 5, 'NO'], \
    ['3 3', 'RIGHT', 1, 'EXIT', 7, 'NO'], ['EXIT', 'NO', 1, 'EXIT', 0, 'NO']]

df = pd.DataFrame(data, columns = ['Start', 'Action', 'Probability', 'End', 'Reward', 'Policy'], \
                  dtype = float) #initial matrix

point_3_0, point_3_1, point_3_2, point_3_3, point_EXIT = 0, 0, 0, 0, 0

gamma = 0.9 #it is a discount factor

for i in range(100): 
    point_3_0 = gamma * max(0.6 * (point_3_1 + 5) + 0.4 * (point_3_2 - 10), point_3_3 + 10)
    point_3_1 = gamma * (point_3_3 + 4)
    point_3_2 = gamma * (0.6 * (point_3_3 + 3) + 0.4 * (point_3_1 + 5))
    point_3_3 = gamma * (point_EXIT + 7)


print(point_3_0, point_3_1, point_3_2, point_3_3, point_EXIT)

But here I have a mistake somewhere and it look like too complex? Could you please help me with this issue?!

gamma = 0.9

class MDP:

    def __init__(self, gamma, table):
        self.gamma = gamma
        self.table = table

    def Action(self, state):
        return self.table[self.table.Start == state].Action.values

    def Probability(self, state):
        return self.table[self.table.Start == state].Probability.values

    def End(self, state):
        return self.table[self.table.Start == state].End.values

    def Reward(self, state):
        return self.table[self.table.Start == state].Reward.values

    def Policy(self, state):
        return self.table[self.table.Start == state].Policy.values

mdp = MDP(gamma = gamma, table = df)

def value_iteration():
    states = mdp.table.Start.values
    actions = mdp.Action
    probabilities = mdp.Probability
    ends = mdp.End
    rewards = mdp.Reward
    policies = mdp.Policy

    V1 = {s: 0 for s in states}
    for i in range(100):
        V = V1.copy()
        for s in states:
            if policies(s) == 'YES':
                V1[s] = gamma * max(rewards(s) + [sum([p * V[s1] for (p, s1) \
                in zip(probabilities(s), ends(s))][actions(s)==a]) for a in set(actions(s))])
            else: 
                sum(probabilities[s] * ends(s))

    return V

value_iteration()

I expect values in every point, but get: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Answer 1

You get the error, because policies(s) = ['YES' 'YES' 'YES'], so it contains 'YES' three times. If you want to check, if all elements in policies(s) are 'YES', simply replace policies(s) == 'YES' with all(x=='YES' for x in policies(s))

If you only want to check for the first element, change to policies(s)[0] == 'YES'

See the Post check if all elements in a list are identical for different approaches.

Answer 2

For the second problem described (assuming (policies(s) == YES).any() fixed the 1st problem) notice that you initialize a regular python list with this expresion

[sum([p * V[s1] for (p, s1) in zip(probabilities(s), ends(s))]

which you then try to access with the indices [actions(s)==a] python lists don't support multiple indexing, and this cause the TypeError you encountered