OpenAI Gym：在動作空間中遍歷所有可能的動作

Question

我想建立一種蠻力方法，在選擇最佳動作之前測試 Gym 動作空間中的所有動作。 是否有任何簡單、直接的方法來獲得所有可能的操作？

具體來說，我的行動空間是

import gym

action_space = gym.spaces.MultiDiscrete([5 for _ in range(4)])

我知道我可以使用action_space.sample()對隨機動作進行采樣，並檢查動作空間中是否包含動作，但我想生成該空間內所有可能動作的列表。

有沒有比一堆 for 循環更優雅（和高性能）的東西？ for 循環的問題是我希望它可以處理任何大小的動作空間，所以我不能硬編碼 4 個 for 循環來遍歷不同的動作。

Answer 1

健身房環境中的動作通常僅由整數表示，這意味着如果您獲得可能動作的總數，則可以創建所有可能動作的數組。

在健身房環境中獲得可能動作總數的方法取決於它具有的動作空間的類型，對於您的情況，它是一個 MultiDiscrete 動作空間，因此屬性 nvec 可以像@Valentin Macé 在這里提到的那樣使用 - ：

>> print(env.action_space.nvec)
array([5, 5, 5, 5], dtype=int64)

注意屬性 nvec 代表 n 向量，因為它的輸出是一個多維向量。 另請注意，該屬性是一個 numpy 數組。

現在我們有了將它轉換成動作列表的數組，假設因為 action_space.sample 函數從 MultiDiscrete action_space 的每個維度返回一個隨機函數的 numpy 數組，即 -：

>> env.action_space.sample() # This does not return a single action but 4 actions for your case since you have a multi discrete action space of length 4.
array([2, 2, 0, 1], dtype=int64)

因此，為了將數組轉換為每個維度中可能操作的列表列表，我們可以使用列表推導式，如下所示 -：

>> [list(range(1, (k + 1))) for k in action_space.nvec]
[[1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5]]

請注意，這可以擴展到任意數量的維度，並且在性能方面也非常高效。

現在，您可以僅使用兩個循環來循環每個維度中可能的操作，如下所示 -：

possible_actions = [list(range(1, (k + 1))) for k in action_space.nvec]
for action_dim in possible_actions :
    for action in action_dim :
        # Find best action.....
        pass

有關相同的更多信息，我希望您也訪問 github 上的這個線程，討論一個有點類似的問題，以防您發現同樣有用。

編輯：因此，根據您@CGFoX的評論，我假設您希望它可以將動作的所有可能組合向量生成為任意數量維度的列表，有點像這樣 -：

>> get_actions()
[[1, 1, 1, 1], [1, 1, 1, 2] ....] # For all possible combinations.

使用遞歸可以像這樣實現相同的效果，並且只有兩個循環，這也可以擴展到所提供的盡可能多的維度。

def flatten(actions) :
    # This function flattens any actions passed somewhat like so -:
    # INPUT -: [[1, 2, 3], 4, 5]
    # OUTPUT -: [1, 2, 3, 4, 5]
    
    new_actions = [] # Initializing the new flattened list of actions.
    for action in actions :
        # Loop through the actions
        if type(action) == list :
            # If any actions is a pair of actions i.e. a list e.g. [1, 1] then
            # add it's elements to the new_actions list.
            new_actions += action
        elif type(action) == int :
            # If the action is an integer then append it directly to the new_actions
            # list.
            new_actions.append(action)
    
    # Returns the new_actions list generated.
    return new_actions

def get_actions(possible_actions) :
    # This functions recieves as input the possibilities of actions for every dimension
    # and returns all possible dimensional combinations for the same.
    # Like so -:
    # INPUT-: [[1, 2, 3, 4], [1, 2, 3, 4]] # Example for 2 dimensions but can be scaled for any.
    # OUTPUT-: [[1, 1], [1, 2], [1, 3] ... [4, 1] ... [4, 4]]
    if len(possible_actions) == 1 :
        # If there is only one possible list of actions then it itself is the
        # list containing all possible combinations and thus is returned.
        return possible_actions
    pairs = [] # Initializing a list to contain all pairs of actions generated.
    for action in possible_actions[0] :
        # Now we loop over the first set of possibilities of actions i.e. index 0
        # and we make pairs of it with the second set i.e. index 1, appending each pair
        # to the pairs list.
        # NOTE: Incase the function is recursively called the first set of possibilities
        # of actions may contain vectors and thus the newly formed pair has to be flattened.
        # i.e. If a pair has already been made in previous generation like so -:
        # [[[1, 1], [2, 2], [3, 3] ... ], [1, 2, 3, 4]]
        # Then the pair formed will be this -: [[[1, 1], 1], [[1, 1], 2] ... ]
        # But we want them to be flattened like so -: [[1, 1, 1], [1, 1, 2] ... ]
        for action2 in possible_actions[1] :
            pairs.append(flatten([action, action2]))
    
    # Now we create a new list of all possible set of actions by combining the 
    # newly generated pairs and the sets of possibilities of actions that have not
    # been paired i.e. sets other than the first and the second.
    # NOTE: When we made pairs we did so only for the first two indexes and not for
    # all thus to do so we make a new list with the sets that remained unpaired
    # and the paired set. i.e.
    # BEFORE PAIRING -: [[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]]
    # AFTER PAIRING -: [[[1, 1], [1, 2] ... ], [1, 2, 3, 4]] # Notice how the third set
    # i.e. the index 2 is still unpaired and first two sets have been paired.
    new_possible_actions = [pairs] + possible_actions[2 : ]
    # Now we recurse the function and call it within itself to make pairs for the
    # left out sets, Note that since the first two sets were combined to form a paired
    # first set now this set will be paired with the third set.
    # This recursion will keep happening until all the sets have been paired to form
    # a single set with all possible combinations.
    possible_action_vectors = get_actions(new_possible_actions)
    # Finally the result of the recursion is returned.
    # NOTE: Only the first index is returned since now the first index contains the
    # paired set of actions.
    return possible_action_vectors[0]

一旦我們定義了這個函數，它就可以與我們之前生成的一組動作可能性一起使用，以獲得所有可能的組合，如下所示：

possible_actions = [list(range(1, (k + 1))) for k in action_space.nvec]
print(get_actions(possible_actions))
>> [[1, 1, 1, 1], [1, 1, 1, 2], [1, 1, 1, 3], [1, 1, 1, 4], [1, 1, 1, 5], `[1, 1, 2, 1], [1, 1, 2, 2], [1, 1, 2, 3], [1, 1, 2, 4], [1, 1, 2, 5], [1, 1, 3, 1], [1, 1, 3, 2], [1, 1, 3, 3], [1, 1, 3, 4], [1, 1, 3, 5], [1, 1, 4, 1], [1, 1, 4, 2], [1, 1, 4, 3], [1, 1, 4, 4], [1, 1, 4, 5], [1, 1, 5, 1], [1, 1, 5, 2], [1, 1, 5, 3], [1, 1, 5, 4], [1, 1, 5, 5], [1, 2, 1, 1], [1, 2, 1, 2], [1, 2, 1, 3], [1, 2, 1, 4], [1, 2, 1, 5], [1, 2, 2, 1], [1, 2, 2, 2], [1, 2, 2, 3], [1, 2, 2, 4], [1, 2, 2, 5], [1, 2, 3, 1], [1, 2, 3, 2], [1, 2, 3, 3], [1, 2, 3, 4], [1, 2, 3, 5], [1, 2, 4, 1], [1, 2, 4, 2], [1, 2, 4, 3], [1, 2, 4, 4], [1, 2, 4, 5], [1, 2, 5, 1], [1, 2, 5, 2], [1, 2, 5, 3], [1, 2, 5, 4], [1, 2, 5, 5], [1, 3, 1, 1], [1, 3, 1, 2], [1, 3, 1, 3], [1, 3, 1, 4], [1, 3, 1, 5], [1, 3, 2, 1], [1, 3, 2, 2], [1, 3, 2, 3], [1, 3, 2, 4], [1, 3, 2, 5], [1, 3, 3, 1], [1, 3, 3, 2], [1, 3, 3, 3], [1, 3, 3, 4], [1, 3, 3, 5], [1, 3, 4, 1], [1, 3, 4, 2], [1, 3, 4, 3], [1, 3, 4, 4], [1, 3, 4, 5], [1, 3, 5, 1], [1, 3, 5, 2], [1, 3, 5, 3], [1, 3, 5, 4], [1, 3, 5, 5], [1, 4, 1, 1], [1, 4, 1, 2], [1, 4, 1, 3], [1, 4, 1, 4], [1, 4, 1, 5], [1, 4, 2, 1], [1, 4, 2, 2], [1, 4, 2, 3], [1, 4, 2, 4], [1, 4, 2, 5], [1, 4, 3, 1], [1, 4, 3, 2], [1, 4, 3, 3], [1, 4, 3, 4], [1, 4, 3, 5], [1, 4, 4, 1], [1, 4, 4, 2], [1, 4, 4, 3], [1, 4, 4, 4], [1, 4, 4, 5], [1, 4, 5, 1], [1, 4, 5, 2], [1, 4, 5, 3], [1, 4, 5, 4], [1, 4, 5, 5], [1, 5, 1, 1], [1, 5, 1, 2], [1, 5, 1, 3], [1, 5, 1, 4], [1, 5, 1, 5], [1, 5, 2, 1], [1, 5, 2, 2], [1, 5, 2, 3], [1, 5, 2, 4], [1, 5, 2, 5], [1, 5, 3, 1], [1, 5, 3, 2], [1, 5, 3, 3], [1, 5, 3, 4], [1, 5, 3, 5], [1, 5, 4, 1], [1, 5, 4, 2], [1, 5, 4, 3], [1, 5, 4, 4], [1, 5, 4, 5], [1, 5, 5, 1], [1, 5, 5, 2], [1, 5, 5, 3], [1, 5, 5, 4], [1, 5, 5, 5], [2, 1, 1, 1], [2, 1, 1, 2], [2, 1, 1, 3], [2, 1, 1, 4], [2, 1, 1, 5], [2, 1, 2, 1], [2, 1, 2, 2], [2, 1, 2, 3], [2, 1, 2, 4], [2, 1, 2, 5], [2, 1, 3, 1], [2, 1, 3, 2], [2, 1, 3, 3], [2, 1, 3, 4], [2, 1, 3, 5], [2, 1, 4, 1], [2, 1, 4, 2], [2, 1, 4, 3], [2, 1, 4, 4], [2, 1, 4, 5], [2, 1, 5, 1], [2, 1, 5, 2], [2, 1, 5, 3], [2, 1, 5, 4], [2, 1, 5, 5], [2, 2, 1, 1], [2, 2, 1, 2], [2, 2, 1, 3], [2, 2, 1, 4], [2, 2, 1, 5], [2, 2, 2, 1], [2, 2, 2, 2], [2, 2, 2, 3], [2, 2, 2, 4], [2, 2, 2, 5], [2, 2, 3, 1], [2, 2, 3, 2], [2, 2, 3, 3], [2, 2, 3, 4], [2, 2, 3, 5], [2, 2, 4, 1], [2, 2, 4, 2], [2, 2, 4, 3], [2, 2, 4, 4], [2, 2, 4, 5], [2, 2, 5, 1], [2, 2, 5, 2], [2, 2, 5, 3], [2, 2, 5, 4], [2, 2, 5, 5], [2, 3, 1, 1], [2, 3, 1, 2], [2, 3, 1, 3], [2, 3, 1, 4], [2, 3, 1, 5], [2, 3, 2, 1], [2, 3, 2, 2], [2, 3, 2, 3], [2, 3, 2, 4], [2, 3, 2, 5], [2, 3, 3, 1], [2, 3, 3, 2], [2, 3, 3, 3], [2, 3, 3, 4], [2, 3, 3, 5], [2, 3, 4, 1], [2, 3, 4, 2], [2, 3, 4, 3], [2, 3, 4, 4], [2, 3, 4, 5], [2, 3, 5, 1], [2, 3, 5, 2], [2, 3, 5, 3], [2, 3, 5, 4], [2, 3, 5, 5], [2, 4, 1, 1], [2, 4, 1, 2], [2, 4, 1, 3], [2, 4, 1, 4], [2, 4, 1, 5], [2, 4, 2, 1], [2, 4, 2, 2], [2, 4, 2, 3], [2, 4, 2, 4], [2, 4, 2, 5], [2, 4, 3, 1], [2, 4, 3, 2], [2, 4, 3, 3], [2, 4, 3, 4], [2, 4, 3, 5], [2, 4, 4, 1], [2, 4, 4, 2], [2, 4, 4, 3], [2, 4, 4, 4], [2, 4, 4, 5], [2, 4, 5, 1], [2, 4, 5, 2], [2, 4, 5, 3], [2, 4, 5, 4], [2, 4, 5, 5], [2, 5, 1, 1], [2, 5, 1, 2], [2, 5, 1, 3], [2, 5, 1, 4], [2, 5, 1, 5], [2, 5, 2, 1], [2, 5, 2, 2], [2, 5, 2, 3], [2, 5, 2, 4], [2, 5, 2, 5], [2, 5, 3, 1], [2, 5, 3, 2], [2, 5, 3, 3], [2, 5, 3, 4], [2, 5, 3, 5], [2, 5, 4, 1], [2, 5, 4, 2], [2, 5, 4, 3], [2, 5, 4, 4], [2, 5, 4, 5], [2, 5, 5, 1], [2, 5, 5, 2], [2, 5, 5, 3], [2, 5, 5, 4], [2, 5, 5, 5], [3, 1, 1, 1], [3, 1, 1, 2], [3, 1, 1, 3], [3, 1, 1, 4], [3, 1, 1, 5], [3, 1, 2, 1], [3, 1, 2, 2], [3, 1, 2, 3], [3, 1, 2, 4], [3, 1, 2, 5], [3, 1, 3, 1], [3, 1, 3, 2], [3, 1, 3, 3], [3, 1, 3, 4], [3, 1, 3, 5], [3, 1, 4, 1], [3, 1, 4, 2], [3, 1, 4, 3], [3, 1, 4, 4], [3, 1, 4, 5], [3, 1, 5, 1], [3, 1, 5, 2], [3, 1, 5, 3], [3, 1, 5, 4], [3, 1, 5, 5], [3, 2, 1, 1], [3, 2, 1, 2], [3, 2, 1, 3], [3, 2, 1, 4], [3, 2, 1, 5], [3, 2, 2, 1], [3, 2, 2, 2], [3, 2, 2, 3], [3, 2, 2, 4], [3, 2, 2, 5], [3, 2, 3, 1], [3, 2, 3, 2], [3, 2, 3, 3], [3, 2, 3, 4], [3, 2, 3, 5], [3, 2, 4, 1], [3, 2, 4, 2], [3, 2, 4, 3], [3, 2, 4, 4], [3, 2, 4, 5], [3, 2, 5, 1], [3, 2, 5, 2], [3, 2, 5, 3], [3, 2, 5, 4], [3, 2, 5, 5], [3, 3, 1, 1], [3, 3, 1, 2], [3, 3, 1, 3], [3, 3, 1, 4], [3, 3, 1, 5], [3, 3, 2, 1], [3, 3, 2, 2], [3, 3, 2, 3], [3, 3, 2, 4], [3, 3, 2, 5], [3, 3, 3, 1], [3, 3, 3, 2], [3, 3, 3, 3], [3, 3, 3, 4], [3, 3, 3, 5], [3, 3, 4, 1], [3, 3, 4, 2], [3, 3, 4, 3], [3, 3, 4, 4], [3, 3, 4, 5], [3, 3, 5, 1], [3, 3, 5, 2], [3, 3, 5, 3], [3, 3, 5, 4], [3, 3, 5, 5], [3, 4, 1, 1], [3, 4, 1, 2], [3, 4, 1, 3], [3, 4, 1, 4], [3, 4, 1, 5], [3, 4, 2, 1], [3, 4, 2, 2], [3, 4, 2, 3], [3, 4, 2, 4], [3, 4, 2, 5], [3, 4, 3, 1], [3, 4, 3, 2], [3, 4, 3, 3], [3, 4, 3, 4], [3, 4, 3, 5], [3, 4, 4, 1], [3, 4, 4, 2], [3, 4, 4, 3], [3, 4, 4, 4], [3, 4, 4, 5], [3, 4, 5, 1], [3, 4, 5, 2], [3, 4, 5, 3], [3, 4, 5, 4], [3, 4, 5, 5], [3, 5, 1, 1], [3, 5, 1, 2], [3, 5, 1, 3], [3, 5, 1, 4], [3, 5, 1, 5], [3, 5, 2, 1], [3, 5, 2, 2], [3, 5, 2, 3], [3, 5, 2, 4], [3, 5, 2, 5], [3, 5, 3, 1], [3, 5, 3, 2], [3, 5, 3, 3], [3, 5, 3, 4], [3, 5, 3, 5], [3, 5, 4, 1], [3, 5, 4, 2], [3, 5, 4, 3], [3, 5, 4, 4], [3, 5, 4, 5], [3, 5, 5, 1], [3, 5, 5, 2], [3, 5, 5, 3], [3, 5, 5, 4], [3, 5, 5, 5], [4, 1, 1, 1], [4, 1, 1, 2], [4, 1, 1, 3], [4, 1, 1, 4], [4, 1, 1, 5], [4, 1, 2, 1], [4, 1, 2, 2], [4, 1, 2, 3], [4, 1, 2, 4], [4, 1, 2, 5], [4, 1, 3, 1], [4, 1, 3, 2], [4, 1, 3, 3], [4, 1, 3, 4], [4, 1, 3, 5], [4, 1, 4, 1], [4, 1, 4, 2], [4, 1, 4, 3], [4, 1, 4, 4], [4, 1, 4, 5], [4, 1, 5, 1], [4, 1, 5, 2], [4, 1, 5, 3], [4, 1, 5, 4], [4, 1, 5, 5], [4, 2, 1, 1], [4, 2, 1, 2], [4, 2, 1, 3], [4, 2, 1, 4], [4, 2, 1, 5], [4, 2, 2, 1], [4, 2, 2, 2], [4, 2, 2, 3], [4, 2, 2, 4], [4, 2, 2, 5], [4, 2, 3, 1], [4, 2, 3, 2], [4, 2, 3, 3], [4, 2, 3, 4], [4, 2, 3, 5], [4, 2, 4, 1], [4, 2, 4, 2], [4, 2, 4, 3], [4, 2, 4, 4], [4, 2, 4, 5], [4, 2, 5, 1], [4, 2, 5, 2], [4, 2, 5, 3], [4, 2, 5, 4], [4, 2, 5, 5], [4, 3, 1, 1], [4, 3, 1, 2], [4, 3, 1, 3], [4, 3, 1, 4], [4, 3, 1, 5], [4, 3, 2, 1], [4, 3, 2, 2], [4, 3, 2, 3], [4, 3, 2, 4], [4, 3, 2, 5], [4, 3, 3, 1], [4, 3, 3, 2], [4, 3, 3, 3], [4, 3, 3, 4], [4, 3, 3, 5], [4, 3, 4, 1], [4, 3, 4, 2], [4, 3, 4, 3], [4, 3, 4, 4], [4, 3, 4, 5], [4, 3, 5, 1], [4, 3, 5, 2], [4, 3, 5, 3], [4, 3, 5, 4], [4, 3, 5, 5], [4, 4, 1, 1], [4, 4, 1, 2], [4, 4, 1, 3], [4, 4, 1, 4], [4, 4, 1, 5], [4, 4, 2, 1], [4, 4, 2, 2], [4, 4, 2, 3], [4, 4, 2, 4], [4, 4, 2, 5], [4, 4, 3, 1], [4, 4, 3, 2], [4, 4, 3, 3], [4, 4, 3, 4], [4, 4, 3, 5], [4, 4, 4, 1], [4, 4, 4, 2], [4, 4, 4, 3], [4, 4, 4, 4], [4, 4, 4, 5], [4, 4, 5, 1], [4, 4, 5, 2], [4, 4, 5, 3], [4, 4, 5, 4], [4, 4, 5, 5], [4, 5, 1, 1], [4, 5, 1, 2], [4, 5, 1, 3], [4, 5, 1, 4], [4, 5, 1, 5], [4, 5, 2, 1], [4, 5, 2, 2], [4, 5, 2, 3], [4, 5, 2, 4], [4, 5, 2, 5], [4, 5, 3, 1], [4, 5, 3, 2], [4, 5, 3, 3], [4, 5, 3, 4], [4, 5, 3, 5], [4, 5, 4, 1], [4, 5, 4, 2], [4, 5, 4, 3], [4, 5, 4, 4], [4, 5, 4, 5], [4, 5, 5, 1], [4, 5, 5, 2], [4, 5, 5, 3], [4, 5, 5, 4], [4, 5, 5, 5], [5, 1, 1, 1], [5, 1, 1, 2], [5, 1, 1, 3], [5, 1, 1, 4], [5, 1, 1, 5], [5, 1, 2, 1], [5, 1, 2, 2], [5, 1, 2, 3], [5, 1, 2, 4], [5, 1, 2, 5], [5, 1, 3, 1], [5, 1, 3, 2], [5, 1, 3, 3], [5, 1, 3, 4], [5, 1, 3, 5], [5, 1, 4, 1], [5, 1, 4, 2], [5, 1, 4, 3], [5, 1, 4, 4], [5, 1, 4, 5], [5, 1, 5, 1], [5, 1, 5, 2], [5, 1, 5, 3], [5, 1, 5, 4], [5, 1, 5, 5], [5, 2, 1, 1], [5, 2, 1, 2], [5, 2, 1, 3], [5, 2, 1, 4], [5, 2, 1, 5], [5, 2, 2, 1], [5, 2, 2, 2], [5, 2, 2, 3], [5, 2, 2, 4], [5, 2, 2, 5], [5, 2, 3, 1], [5, 2, 3, 2], [5, 2, 3, 3], [5, 2, 3, 4], [5, 2, 3, 5], [5, 2, 4, 1], [5, 2, 4, 2], [5, 2, 4, 3], [5, 2, 4, 4], [5, 2, 4, 5], [5, 2, 5, 1], [5, 2, 5, 2], [5, 2, 5, 3], [5, 2, 5, 4], [5, 2, 5, 5], [5, 3, 1, 1], [5, 3, 1, 2], [5, 3, 1, 3], [5, 3, 1, 4], [5, 3, 1, 5], [5, 3, 2, 1], [5, 3, 2, 2], [5, 3, 2, 3], [5, 3, 2, 4], [5, 3, 2, 5], [5, 3, 3, 1], [5, 3, 3, 2], [5, 3, 3, 3], [5, 3, 3, 4], [5, 3, 3, 5], [5, 3, 4, 1], [5, 3, 4, 2], [5, 3, 4, 3], [5, 3, 4, 4], [5, 3, 4, 5], [5, 3, 5, 1], [5, 3, 5, 2], [5, 3, 5, 3], [5, 3, 5, 4], [5, 3, 5, 5], [5, 4, 1, 1], [5, 4, 1, 2], [5, 4, 1, 3], [5, 4, 1, 4], [5, 4, 1, 5], [5, 4, 2, 1], [5, 4, 2, 2], [5, 4, 2, 3], [5, 4, 2, 4], [5, 4, 2, 5], [5, 4, 3, 1], [5, 4, 3, 2], [5, 4, 3, 3], [5, 4, 3, 4], [5, 4, 3, 5], [5, 4, 4, 1], [5, 4, 4, 2], [5, 4, 4, 3], [5, 4, 4, 4], [5, 4, 4, 5], [5, 4, 5, 1], [5, 4, 5, 2], [5, 4, 5, 3], [5, 4, 5, 4], [5, 4, 5, 5], [5, 5, 1, 1], [5, 5, 1, 2], [5, 5, 1, 3], [5, 5, 1, 4], [5, 5, 1, 5], [5, 5, 2, 1], [5, 5, 2, 2], [5, 5, 2, 3], [5, 5, 2, 4], [5, 5, 2, 5], [5, 5, 3, 1], [5, 5, 3, 2], [5, 5, 3, 3], [5, 5, 3, 4], [5, 5, 3, 5], [5, 5, 4, 1], [5, 5, 4, 2], [5, 5, 4, 3], [5, 5, 4, 4], [5, 5, 4, 5], [5, 5, 5, 1], [5, 5, 5, 2], [5, 5, 5, 3], [5, 5, 5, 4], [5, 5, 5, 5]]

EDIT-2 ：我已經修復了一些以前返回嵌套列表的代碼，現在返回的列表是帶有成對的列表，而不是嵌套在另一個列表中。

EDIT-3-：修正了我的拼寫錯誤。

Answer 2

還可以使用下面的函數，分別在觀察或動作空間中制作所有狀態或動作的明確列表。

def get_space_list(space):

    """
    Converts gym `space`, constructed from `types`, to list `space_list`
    """

    # -------------------------------- #

    types = [
        gym.spaces.multi_binary.MultiBinary,
        gym.spaces.discrete.Discrete,
        gym.spaces.multi_discrete.MultiDiscrete,
        gym.spaces.dict.Dict,
        gym.spaces.tuple.Tuple,
    ]

    if type(space) not in types:
        raise ValueError(f'input space {space} is not constructed from spaces of types:' + '\n' + str(types))

    # -------------------------------- #

    if type(space) is gym.spaces.multi_binary.MultiBinary:
        return [
            np.reshape(np.array(element), space.n)
            for element in itertools.product(
                *[range(2)] * np.prod(space.n)
            )
        ]

    if type(space) is gym.spaces.discrete.Discrete:
        return list(range(space.n))

    if type(space) is gym.spaces.multi_discrete.MultiDiscrete:
        return [
            np.array(element) for element in itertools.product(
                *[range(n) for n in space.nvec]
            )
        ]

    if type(space) is gym.spaces.dict.Dict:

        keys = space.spaces.keys()
        
        values_list = itertools.product(
            *[get_space_list(sub_space) for sub_space in space.spaces.values()]
        )

        return [
            {key: value for key, value in zip(keys, values)}
            for values in values_list
        ]

        return space_list

    if type(space) is gym.spaces.tuple.Tuple:
        return [
            list(element) for element in itertools.product(
                *[get_space_list(sub_space) for sub_space in space.spaces]
            )
        ]

    # -------------------------------- #