[英]How to set a constant as a weight in a keras model?
I build my model using tf.keras.layers.Dense
. 我使用tf.keras.layers.Dense
构建模型。 In the first layer of my model I want some weights to be constant Zero. 在模型的第一层中,我希望一些权重为零。 As in the gradient calculation these weights should be get a gradient = zero (as the last term in the chain rule corresponds to the weight, which is 0 for a constant). 在梯度计算中,这些权重应为梯度= 0(因为链式规则中的最后一项与权重相对应,常数为0)。 This is my approach so far: 到目前为止,这是我的方法:
import tensorflow as tf
import tensorflow.contrib.eager as tfe
import numpy as np
tf.enable_eager_execution()
model = tf.keras.Sequential([
tf.keras.layers.Dense(2, activation=tf.sigmoid, input_shape=(2,)),
tf.keras.layers.Dense(2, activation=tf.sigmoid)
])
weights=[np.array([[tf.constant(0), 0.25],[0.2,0.3]]),np.array([0.35,0.35]),np.array([[0.4,0.5],[0.45, 0.55]]),np.array([0.6,0.6])]
model.set_weights(weights)
def loss(model, x, y):
y_ = model(x)
return tf.losses.mean_squared_error(labels=y, predictions=y_)
def grad(model, inputs, targets):
with tf.GradientTape() as tape:
loss_value = loss(model, inputs, targets)
return loss_value, tape.gradient(loss_value, model.trainable_variables)
But in the gradient calculation the weight tf.constant(0) has a gradient not equal zero. 但是在梯度计算中,权重tf.constant(0)的梯度不等于零。 Do I have an understanding problem? 我有理解上的问题吗?
How can I set a weight(or some weights) in a layer( not all weights in one layer) to a constant value (which should not change during training)? 我怎样才能在一个层( 未在一层所有的权重) 的重量( 或权重)设置为一个固定值(这不应该训练过程中更改)?
My answer is based on the CustomConnected
layer from this answer . 我的答案基于此答案的CustomConnected
层。 As I said in a comment, when you multiply a weight w_ij
by c_ij=0
via the connections matrix, the gradient of the loss with respect to that weight becomes zero as well (since the last factor in the chain rule corresponds to c_ij=0
). 正如我在评论中所说,当通过连接矩阵将权重w_ij
乘以c_ij=0
时,损失相对于该权重的梯度也将变为零(因为链式规则中的最后一个因子对应于c_ij=0
)。
Here is a minimal example in Keras: 这是Keras中的一个最小示例:
# Using CustomConnected from:
# https://stackoverflow.com/questions/50290769/specify-connections-in-nn-in-keras
import tensorflow as tf
import numpy as np
tf.enable_eager_execution()
# Define model
inp = tf.keras.layers.Input(shape=(2,))
c = np.array([[1., 1.], [1., 0.]], dtype=np.float32)
h = CustomConnected(2, c)(inp)
model = tf.keras.models.Model(inp, h)
# Set initial weights and compile
w = [np.random.rand(2, 2) * c]
model.set_weights(w)
model.compile(tf.train.AdamOptimizer(), 'mse')
# Check gradients
x = tf.constant(np.random.rand(10, 2), dtype=tf.float32)
y = np.random.rand(10, 2)
with tf.GradientTape() as tape:
loss_value = tf.losses.mean_squared_error(labels=y, predictions=model(x))
grad = tape.gradient(loss_value, model.trainable_variables)
print('Gradients: ', grad[0])
Note that I set c[1,1]=0
so the gradient corresponding to weight w[1,1]
is 0 regardless of the input. 请注意,我将c[1,1]=0
设置为与输入w[1,1]
相对应的梯度为0,与输入无关。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.