如何扩展神经网络层？

Question

I am trying to write an expandable neural network described in this paper: Lifelong Learning with Dynamically Expandable Networks (Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang, Aug 2017).

Now consider I have such a LSTM layer:

tf.contrib.rnn.LSTMCell(128, state_is_tuple=True)

Say I want to leave the gates for now and just want to expand the neurons of this LSTM layer from 128 to 256 while retaining the previous weights and make the new weights 0.000001 which makes it insignificant for the following layers. How can I do that?

Also, is it possible to change the input size and retain the weights by assigning zero weights to the new features at the beginning? For example, if I have input like this:

inputs = tf.placeholder(tf.float32, [None, 30, 2])

and I want to change it to:

inputs = tf.placeholder(tf.float32, [None, 30, 5])

Then, what should I do to my layers to load the previously trained weights and padding zeros to fit the input shape?

An answer to any other types of layers will be very much appreciated as well. Literally, any help will be appreciated.

Answer 1

1. 新的权重应初始化为随机值