如何在TensorFlow RNN中检索中间状态

Question

I am running an RNN on a signal in fixed-size segments. The following code allows me to preserve the final state of the previous batch to initialize the initial state of the next batch.

rnn_outputs, final_state = tf.contrib.rnn.static_rnn(cell, rnn_inputs, initial_state=init_state)

This works when the batches are non-overlapping. For example, my first batch processes samples 0:124 and final_state is the state after this processing. Then, the next batch processes samples 124:256, setting init_state to final_state .

My question is how to retrieve an intermediary state when the batches are overlapping. First, I process samples 0:124, then 10:134, 20:144, so the hop size is 10. I would like to retrieve not the final_state but the state after processing 10 samples.

Is it possible in TF to keep the intermediary state? The documentation shows that the return value consists only of the final state.

The image shows the issue I am facing due to state discontinuity. In my program, the RNN segment length is 215 and the hop length is 20.

Update: the easiest turned out to be what David Parks described:

rnn_outputs_one, mid_state = tf.contrib.rnn.static_rnn(cell, rnn_inputs_one, initial_state=rnn_tuple_state)
rnn_outputs_two, final_state = tf.contrib.rnn.static_rnn(cell, rnn_inputs_two, initial_state=mid_state)
rnn_outputs = rnn_outputs_one + rnn_outputs_two

and

prev_state = sess.run(mid_state)

Now, after just a few iterations, the results look much better.

Answer 1

In tensorflow the only thing that is kept after returning from a call to sess.run are variables. You should create a variable for the state, then use tf.assign to assign the result from your RNN cell to that variable. You can then use that Variable in the same way as any other tensor.

If you need to initialize the variable to something other than 0 you can call sess.run once with a placeholder and tf.assign specifically to setup the variable.

---

Added detail:

If you need an intermediate state, let's say you ran for timesteps 0:124 and you want step 10, you should split that up into 2 RNN cells, one that processes the first 10 timesteps and the second that continues processing the next 114 timesteps. This shouldn't affect training and back propagation as long as you use the same cell (LSTM or other cell) in both static_rnn functions. The cell is where your weights are defined, and that has to remain constant. Your gradient will flow backwards through the second cell and then finally the first appropriately.

Answer 2

So, I came here looking for an answer earlier, but I ended up creating one.

Similar to above posters about making it assignable...

When you build your graph, make a list of sequence placeholders like..

my_states = [None] * int(sequence_length + 1)

my_states[0] = cell.zero_state()

for step in steps:
    cell_out, my_states[step+1] = cell( ) 

Then outside of your graph after the sess.run() you say

new_states = my_states[1:]

model.my_states = new_states

This situation is for stepping 1 timestep at a time, but it could easily be made for steps of 10. Just slice the list of states after sess.run() and make those the initial states.

Good luck!