How to reuse RNN in TensorFlow

Question

I want to implement a model like DSSM (Deep Semantic Similarity Model).

I want to train one RNN model and use this model to get three hidden vector for three different inputs, and use these hidden vector to compute loss function.

I try to code in a variable scope with reuse=None like:

gru_cell = tf.nn.rnn_cell.GRUCell(size)
gru_cell = tf.nn.rnn_cell.DropoutWrapper(gru_cell,output_keep_prob=0.5)
cell = tf.nn.rnn_cell.MultiRNNCell([gru_cell] * 2, state_is_tuple=True)

embedding = tf.get_variable("embedding", [vocab_size, wordvec_size])
inputs = tf.nn.embedding_lookup(embedding, self._input_data)
inputs = tf.nn.dropout(inputs, 0.5)
with tf.variable_scope("rnn"):
    _, self._states_2 = rnn_states_2[config.num_layers-1] = tf.nn.dynamic_rnn(cell, inputs, sequence_length=self.lengths, dtype=tf.float32)
    self._states_1 = rnn_states_1[config.num_layers-1]
with tf.variable_scope("rnn", reuse=True):
    _, rnn_states_2 = tf.nn.dynamic_rnn(cell,inputs,sequence_length=self.lengths,dtype=tf.float32)
    self._states_2 = rnn_states_2[config.num_layers-1]

I use the same inputs and reuse the RNN model, but when I print 'self_states_1' and 'self_states_2', these two vectors are different.

I use with tf.variable_scope("rnn", reuse=True): to compute 'rnn_states_2' because I want to use the same RNN model like 'rnn_states_1'.

But why I get different hidden vectors with the same inputs and the same model?

Where did i go wrong?

Thanks for your answering.

Update: I find the reason may be the 'tf.nn.rnn_cell.DropoutWrapper' , when I remove the drop out wrapper, the hidden vectors are same, when I add the drop out wrapper, these vector become different.

So, the new question is :

How to fix the part of vector which be 'dropped out' ? By setting the 'seed' parameter ?

When training a DSSM, should I fix the drop out action ?

Answer 1

If you structure your code to use tf.contrib.rnn.DropoutWrapper , you can set variational_recurrent=True in your wrapper, which causes the same dropout mask to be used at all steps, ie the dropout mask will be constant. Is that what you want?

Setting the seed parameter in tf.nn.dropout will just make sure that you get the same sequence of dropout masks every time you run with that seed. That does not mean the dropout mask will be constant , just that you'll always see the same dropout mask at a particular iteration. The mask will be different for every iteration.