Is RepeatVector layer the same as Concatenate layer?

Question

Here was my code:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, RepeatVector

Model = Sequential([
              LSTM(2, input_shape=(3,2)),
              RepeatVector(5)])

inputs = [[[1,2], [3,4], [5,6]]]
print(Model.predict(inputs))

which produces the output:

[[[0.03203796 0.00486411]
[0.03203796 0.00486411]
[0.03203796 0.00486411]
[0.03203796 0.00486411]
[0.03203796 0.00486411]]]

Now, it seems that adding Repeat Vector just repeated the same output n times. If RepeatVector just repeats the output n times, does this mean that RepeatVector(n) is the same as Concatenating the output to itself n times?

Answer 1

Those two layers, Concatenate and RepeaVector are syntactic sugar over the tensorflow ops tf.concat and tf.repeat respectively.

Those 2 ops are there to mimic the numpy operations numpy.concatenate and numpy.repeat .

Those 2 ops are different:

• repeat will repeat each element of the Tensor n times
• concat will concat the tensor with itself

import numpy as np
import tensorflow as tf
a = np.arange(3)
>>> a
array([0, 1, 2])
>>> tf.repeat(a,3)
<tf.Tensor: shape=(9,), dtype=int64, numpy=array([0, 0, 0, 1, 1, 1, 2, 2, 2])>
>>> tf.concat((a,)*3,axis=0)
<tf.Tensor: shape=(9,), dtype=int64, numpy=array([0, 1, 2, 0, 1, 2, 0, 1, 2])>

tf.repeat was introduced in a feature request on github

Answer 2

According to the docs of RepeatVector and concatenate , one of the differences is the output shape.

For RepeatVector , the output shape is: (num_samples, n, features)

For concatenate , the output shape is: (num_samples, features*n) (if axis=-1 )

Let's visualize that. Here's some dummy input:

import tensorflow as tf

x = tf.reshape(tf.range(1, 10), (3, 3))

<tf.Tensor: shape=(3, 3), dtype=int32, numpy=
array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]])>

Let's repeat the vector:

tf.keras.layers.RepeatVector(n=3)(x)

<tf.Tensor: shape=(3, 3, 3), dtype=int32, numpy=
array([[[1, 2, 3],
        [1, 2, 3],
        [1, 2, 3]],
       [[4, 5, 6],
        [4, 5, 6],
        [4, 5, 6]],
       [[7, 8, 9],
        [7, 8, 9],
        [7, 8, 9]]])>

Let's concatenate:

tf.keras.layers.concatenate([x, x, x])

<tf.Tensor: shape=(3, 9), dtype=int32, numpy=
array([[1, 2, 3, 1, 2, 3, 1, 2, 3],
       [4, 5, 6, 4, 5, 6, 4, 5, 6],
       [7, 8, 9, 7, 8, 9, 7, 8, 9]])>

The other difference is of course, the conceptual difference between repeating and concatenating.

Repeating:

[0, 0, 0, 1, 1, 1, 2, 2, 2]

Concatenating:

[0, 1, 2, 0, 1, 2, 0, 1, 2]