了解Tensorflow中权重和偏差的初始化

Question

Presently I am using stacked LSTM and FC layer at last.

tf.contrib.rnn.BasicLSTMCell
tf.contrib.rnn.MultiRNNCell
tf.nn.dynamic_rnn
tf.contrib.layers.fully_connected

According to my understanding, if I try to use any architecture defined under

tf.nn

class then weight initialization like

W2 = tf.Variable(np.random.rand(state_size, num_classes),dtype=tf.float32)
b2 = tf.Variable(np.zeros((1,num_classes)), dtype=tf.float32) 

needs to be done and tf.matmul has to be used.

But for class under

tf.contrib

weight initialization happens automatically and tf.matmul may not be required.

Is my understanding correct? Please let me know

Answer 1

In your case the creation of weight and bias variables happens inside the cell, and then inside the fully connected layer. So, there is no need to define them explicitly. Also, when you are building the graph, Tensorflow doesn't initialize any of your variables. After, before you start executing nodes in the graph, you have you initialize the variables in the graph beforehand. Have a look:

# Some input
inputs = np.ones((5,4,3), dtype=np.float32)
# Cleaning the default graph before building it
tf.reset_default_graph()
# Creating a MultiRNNCell (The cell itself creates the weight and bias matrices)
cells = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicLSTMCell(10) for _ in range(2)])
# A wrapper around the cells to ensure the recurrent relation
hidden_states, cell_state = tf.nn.dynamic_rnn(cell=cells, inputs=inputs, dtype=tf.float32)
# Fully connected layer (What it does is create a W and b matrices and performs tf.matmul and then tf.add)
logits = tf.contrib.layers.fully_connected(hidden_states[-1], num_outputs=2, activation_fn=None)
# A new session
sess = tf.Session()
# All variables that were created above (implicitly, by creating a cell and a fully connected layer) will be initialized
sess.run(tf.global_variables_initializer())
# Executing the last node in the graph
sess.run(logits)