In Tensorflow, what is the difference between the returned 'output' and 'h' of state tuple (c, h) in LSTMCell?

Question

I've searched across many tutorials/blogs/guides and official Tensorflow documentation to understand this. For example, see below lines:

lstm = tf.nn.rnn_cell.LSTMCell(512)
output, state_tuple = lstm(current_input, last_state_tuple)

Now if I unpack state,

last_cell_memory, last_hidden_state =  state_tuple

Both output and last_hidden_state have exact same dimensions of [batch_size, 512]. Can both be used interchangeably? I mean, can I do this? :

last_state_tuple= last_cell_memory, output 

and then feed last_state_tuple in lstm?

Answer 1

Jacques's answer is correct, but it doesn't mention an important point: the state of LSTM layer almost always equals to the output. The difference becomes important when the chain of LSTM cells is long and not all input sequences have equal length (and hence are padded). That's when you should distinguish the state and output.

See the runnable example in my answer on a similar question (it uses BasicRNNCell , but you'll get the same result with LSTMCell ).

Answer 2

Yes, the second element of the state is the same as the output.

From https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/LSTMStateTuple

Stores two elements: (c, h), in that order. Where c is the hidden state and h is the output.

Also to verify experimentally:

import tensorflow as tf
from numpy import random as rng
lstm = tf.nn.rnn_cell.LSTMCell(10)
inp = tf.placeholder(tf.float32, shape=(1, 10))
stt = tf.placeholder(tf.float32, shape=(1, 10))
hdd = tf.placeholder(tf.float32, shape=(1, 10))
out = lstm(inp, (stt, hdd))
sess = tf.InteractiveSession()
init = tf.global_variables_initializer()
sess.run(init)
a = rng.randn(1, 10)
b = rng.randn(1, 10)
c = rng.randn(1, 10)
output = sess.run(out, {inp: a, stt: b, hdd: c})
assert (output[0] == output[1][1]).all()