ValueError: Shapes (?, 83) and (?, 128) are incompatible

Question

I'm getting the following error when trying to create a LSTM using TensorFlow:
ValueError: Shapes (?, 83) and (?, 128) are incompatible

The input to the model have the following shape:
(batch_size, time, features) / (50, 68, 83)

Here is the relevant code for the model:

x_text = tf.placeholder(tf.float32, [None, *text.shape[1:]])

cells = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.ResidualWrapper(
    tf.contrib.rnn.BasicLSTMCell(num_units=units))
    for units in [128, 256]
])

text_outputs, text_state = tf.nn.dynamic_rnn(
    cell=cells,
    inputs=x_text,
    dtype=tf.float32,
)

I've tried for so long to figure out what's wrong, but I can't. I've search the entire internett (no, really!) and no one seems to be having the same problem where shapes (?, a) and (?, b) are the problem, but rather all other combinations where the solutions has not helped.

Oh - and here's the stack trace:

Traceback (most recent call last):
  File "residual_hierachical_rnn.py", line 97, in <module>
    dtype=tf.float32,
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py", line 627, in dynamic_rnn
    dtype=dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py", line 824, in _dynamic_rnn_loop
    swap_memory=swap_memory)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3224, in while_loop
    result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2956, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2893, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3194, in <lambda>
    body = lambda i, lv: (i + 1, orig_body(*lv))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py", line 795, in _time_step
    (output, new_state) = call_cell()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn.py", line 781, in <lambda>
    call_cell = lambda: cell(input_t, state)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 232, in __call__
    return super(RNNCell, self).__call__(inputs, state)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/base.py", line 717, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 1292, in call
    cur_inp, new_state = cell(cur_inp, cur_state)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 1168, in __call__
    res_outputs = (self._residual_fn or default_residual_fn)(inputs, outputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 1166, in default_residual_fn
    nest.map_structure(assert_shape_match, inputs, outputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py", line 375, in map_structure
    structure[0], [func(*x) for x in entries])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py", line 375, in <listcomp>
    structure[0], [func(*x) for x in entries])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/rnn_cell_impl.py", line 1163, in assert_shape_match
    inp.get_shape().assert_is_compatible_with(out.get_shape())
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py", line 844, in assert_is_compatible_with
    raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (?, 83) and (?, 128) are incompatible

Thank you so much in advance for any help!!!

Answer 1

The unit-size of your LSTM cells should match the number of features. To fix this, use [83, 256] .

Also, I'm aware that the connections between the LSTM layers are fully connected, but from what I've been told it's better to keep the unit size in the layers consistent to make things less confusing. In other words, consider using [83, 83] for your unit-sizes.

Answer 2

As MPKenning says, your LSTM cells should match your input features.

But the fact that you're using a ResidualWrapper forces you to keep the same depth because what it does is summing up the inputs and the outputs of the cells.

If you remove the ResidualWrapper it should work with [83, 256] :

cells = tf.contrib.rnn.MultiRNNCell([
    tf.contrib.rnn.BasicLSTMCell(num_units=units)
    for units in [83, 256]
])