类型错误：调用 tensorflow 优化器 `apply_gradients` 时第一个参数必须是可调用的

Question

我希望有人能帮我解决这个让我发疯好几天的问题。 我正在构建一些受此keras 示例启发的东西。 我正在尝试手动计算 a.network 的梯度，但我无法弄清楚我做错了什么。 这是 model 的定义

inputs = layers.Input(shape=(state_dim,))
layer1 = layers.Dense(l1_dim, activation="relu")(inputs)
ayer2 = layers.Dense(l2_dim, activation="relu")(layer1)
action = layers.Dense(num_actions, activation="softmax")(layer2)
critic = layers.Dense(1, activation=None)(layer2)

model = keras.Model(inputs=inputs, outputs=[critic, action])
# model.compile(optimizer=keras.optimizers.Adam(learning_rate=learning_rate))
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)

然后我有我的训练循环，给定集合（状态，动作，奖励，终端，state_）：

state = tf.convert_to_tensor([state], dtype=tf.float32)
state_ = tf.convert_to_tensor([state_], dtype=tf.float32)
reward = tf.convert_to_tensor(reward, dtype=tf.float32)  # not fed to NN
with tf.GradientTape(persistent=True) as tape:
     state_value, probs = model(state)
     state_value_, _ = model(state_)
     state_value = tf.squeeze(state_value)
     state_value_ = tf.squeeze(state_value_)

     action_probs = tfp.distributions.Categorical(probs=probs)
     log_prob = action_probs.log_prob(action)

     delta = reward + self.gamma * state_value_ * (1 - int(terminal)) - state_value
     actor_loss = -log_prob * delta
     critic_loss = delta ** 2
     total_loss = actor_loss + critic_loss

gradient = tape.gradient(total_loss, model.trainable_variables)
optimizer.apply_gradients(zip(gradient, model.trainable_variables))

但是在我的最后一行代码中，调用optimizer.apply_gradients时出现以下错误：

Traceback (most recent call last):
  File "/Users/maccheroni/.virtualenvs/rl_gym/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py", line 639, in apply_gradients
    self._create_all_weights(var_list)
  File "/Users/maccheroni/.virtualenvs/rl_gym/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py", line 829, in _create_all_weights
    self._create_hypers()
  File "/Users/maccheroni/.virtualenvs/rl_gym/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py", line 977, in _create_hypers
    self._hyper[name] = self.add_weight(
  File "/Users/maccheroni/.virtualenvs/rl_gym/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py", line 1192, in add_weight
    variable = self._add_variable_with_custom_getter(
  File "/Users/maccheroni/.virtualenvs/rl_gym/lib/python3.9/site-packages/tensorflow/python/training/tracking/base.py", line 816, in _add_variable_with_custom_getter
    new_variable = getter(
  File "/Users/maccheroni/.virtualenvs/rl_gym/lib/python3.9/site-packages/keras/engine/base_layer_utils.py", line 106, in make_variable
    init_val = functools.partial(initializer, shape, dtype=dtype)
TypeError: the first argument must be callable

我真的不明白为什么，因为我已经阅读了很多教程，遵循了很多示例，他们似乎都以这种方式使用这个 function。

Answer 1

我也有同样的错误并找到了解决方案。 在我的例子中，优化器的初始化：

optimizer = keras.optimizers.Adam(learning_rate=learning_rate)

使用的变量learning_rate是None 。 用数字或简单地初始化：

optimizer = keras.optimizers.Adam()

解决了这个问题。 在你的情况下，不清楚learning_rate是什么，但你可以检查一下。