並行運行模型的多個克隆

Question

因此，我正在嘗試使用Evolution Strategy實施強化學習算法。

原理是將您的原始模型克隆N次（比如說100次），對這100個克隆應用一些噪聲，運行它們，檢查哪些克隆產生了最佳效果，然后使用它來更新原始模型。

現在，我試圖將每個克隆放入不同的線程中，然后並行運行它們。

這是我的工人班：

class WorkerThread(Thread):

    def __init__(self, action_dim, img_dim, sigma, sess):
        Thread.__init__(self)
        #sess = tf.Session()
        self.actor = ActorNetwork(sess, action_dim, img_dim)
        self.env = Environment()
        self.reward = 0
        self.N = {}
        self.original_model = None
        self.sigma = sigma

    def setActorModel(self, model):
        self.original_model = model

    def run(self):
        k = 0
        for l in self.actor.model.layers:
            if len(np.array(l.get_weights())) > 0:
                # First generate some noise
                shape = (np.array(l.get_weights()[0])).shape
                if len(shape) == 2:
                    self.N[k] = np.random.randn(shape[0], shape[1])
                else:
                    self.N[k] = np.random.randn(shape[0], shape[1], shape[2], shape[3])
                # 2nd set weights using original model's weights and noise
                la = self.original_model.layers[k]
                self.actor.model.layers[k].set_weights((la.get_weights()[0] + self.sigma * self.N[k], la.get_weights()[1]))

            k += 1

        ob = self.env.reset()

        while True:
            action = self.actor.predict(np.reshape(ob['image'], (1, 480, 480, 3)))
            ob = self.env.step(action[0])

            if ob['done']:
                self.reward = ob['reward']
                break

因此，每個工作線程都有自己的模型，在運行時，我使用原始模型的權重設置權重。

那時我得到以下錯誤

  File "/usr/local/lib/python3.6/site-packages/keras/engine/topology.py", line 1219, in set_weights
    K.batch_set_value(weight_value_tuples)
  File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2365, in batch_set_value
    assign_op = x.assign(assign_placeholder)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 594, in assign
    return state_ops.assign(self._variable, value, use_locking=use_locking)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 276, in assign
    validate_shape=validate_shape)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 59, in assign
    use_locking=use_locking, name=name)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 350, in _apply_op_helper
    g = ops._get_graph_from_inputs(_Flatten(keywords.values()))
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5055, in _get_graph_from_inputs
    _assert_same_graph(original_graph_element, graph_element)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 4991, in _assert_same_graph
    original_item))
ValueError: Tensor("Placeholder:0", shape=(5, 5, 3, 24), dtype=float32) must be from the same graph as Tensor("conv2d_11/kernel:0", shape=(5, 5, 3, 24), dtype=float32_ref).

在上面的代碼示例中，我在所有線程中使用相同的tensorflow會話。 我嘗試為每個會話創建一個不同的會話，但出現相同的錯誤。

我對tensorflow知之甚少，有人知道如何解決這個問題嗎？

Answer 1

您需要在所有線程中使用相同的圖。 在主線程中創建一個tf.Graph（）並將每個線程的函數包裝在“ with my_graph.as_default（）：”中。