[英]How do I train only the new weights?
Consider a (dense) layer of 10 units. 考虑10个单元的(密集)层。 I would like now to add another (single) unit to this layer.
我现在想在此层添加另一个(单个)单元。 But how do I make sure that the previous 10 weights are not trained, and only the new one gets trained?
但是,如何确保不训练之前的10个权重,而仅训练新的10个权重?
You can pass a list of variables to different optimizers. 您可以将变量列表传递给不同的优化器。 One for training the first 10 units the other one for training the other one in conjunction with a
tf.cond()
so you can have two branches in your graph for when you want to use only 10 or 11 neurons. 一个用于训练前10个单元,另一个用于与
tf.cond()
一起训练另一个单元,因此,当您只想使用10或11个神经元时,可以在图形中拥有两个分支。 You can also play with tf.stop_gradient()
it really depends on your use case. 您还可以使用
tf.stop_gradient()
这实际上取决于您的用例。
Without more informations it is difficult to answer but somewhere along the lines of: 没有更多的信息,很难回答,但遵循以下方式:
import tensorflow as tf
choose_branch=tf.placeholder(tf.bool)
w_old=tf.Variable(np.random.normal(size=(1,10)).astype("float32"))
b=tf.Variable(np.zeros((1)).astype("float32"))
x=tf.placeholder(tf.float32,size=[None,1])
target=tf.placeholder(tf.float32,size=[None,1])
hidden_old=tf.nn.relu(tf.matmul(x,w_old)+b)
w_proj_old=tf.Variable(np.random.normal(size=(10,1)).astype("float32")
y_old=tf.matmul(hidden_old, w_proj_old)
cost_old=tf.reduce_mean(tf.square(y_old-target))
w_plus=tf.Variable(np.random.normal(size=(1,1)).astype("float32"))
w_proj_plus=tf.Variable(np.random.normal(size=(1,1)).astype("float32")
w_proj_new=tf.concat([w_proj_old,w_proj_plus],axis=1)
w_new=tf.concat([w_old,w_plus],axis=1)
hidden_new=tf.nn.relu(tf.matmul(x,w_new,axis=1))+b))
y_new=tf.matmul(hidden_new, w_proj_new)
cost_new=tf.reduce_mean(tf.square(y_new-target))
opt_old=tf.train.GradientDescentOptimizer(0.001)
opt_new=tf.train.GradientDescentOptimizer(0.0001)
train_step_old=opt_old.minimize(var_list=[w_old,b,w_proj_old,b])
train_step_new=opt_new.minimize(var_list=[w_plus,w_proj_plus])
y=tf.cond(choose_branch,lambda: y_old,lambda: y_new)
Careful the code was not tested. 小心代码未经测试。
To do this kind of advanced operations I find it easier to switch to the lower API of tensorflow. 为了进行这种高级操作,我发现切换到较低的Tensorflow API更容易。
A fully connected layer is usually defined as a matrix multiplication. 通常将完全连接的层定义为矩阵乘法。 For example, suppose your previous layer has
128
features, and you want to implement a fully connected layer with 256
features. 例如,假设您的上一层具有
128
功能,并且您想要实现一个具有256
功能的完全连接的层。 You could write 你可以写
batch_size = 128
num_in = 128
num_out = 256
h = tf.zeros((batch_size, num_in)) # actually, the output of the previous layer
# create the weights for the fully connected layer
w = tf.get_variable('w', shape=(num_in, num_out))
# compute the output
y = tf.matmul(h, w)
# -- feel free to add biases and activation
Now let's suppose you have trained w
and want to add some extra neurons to this layer. 现在,让我们假设您已经训练了
w
并且想要在此层添加一些额外的神经元。 You could create an extra variable holding the extra weights, and concatenate it with the existing one. 您可以创建一个包含额外权重的额外变量,并将其与现有变量连接起来。
num_out_extra = 10
# now w is set to trainable=False, we don't want its values to change
w = tf.get_variable('w', shape=(num_in, num_out), trainable=False)
# our new weights
w_extra = tf.get_variable('w_extra', shape=(num_in, num_out_extra))
w_total = tf.concat([w, w_extra], axis=-1)
y = tf.matmul(h, w_total)
# now y has 266 features
You will need to initialize all the weights one way or the other of course. 当然,您将需要以一种或另一种方式初始化所有权重。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.