Concatenating features to word embedding at the input layer during run time

Question

Suppose I obtain an input matrix after embedding lookup which looks like:

[ [[ 0.5, 0.25, 0.47, 0.86 ], [ 0.8. 0.12, 0.63, 0.97 0.8. 0.12, 0.63, 0.97 ], [ 0.7, 0.47, 0.32, 0.01 ]], ..., [[...]] ] i.,e each embedding is of dim = 4 and sentence length be 3 as given in the mentioned case.

How can we append a feature vector of dim say 2, corresponding to each word in the sentence dynamically (ie, during run time) using the placeholder in Tensorflow/TFLearn or Theano? So final input will be of dim = embedding_dim + feature_dim.

PS: Input matrix is a 3D tensor of shape [xyz], x = No. of sentences in batch, y = No. of words in the sentences (including padding). z = Embedding dimension. Final shape would be [xy z+2].

Answer 1

In Tensorflow you can create a placeholder of the desired shape [x, y, 2] and then concatenate it to the input Tensor using tf.concat. Assuming 'inputs' is your [x, y, z] embedding Tensor, you can do something like this:

features = tf.placeholder(tf.float32, [x, y, 2])
new_inputs = tf.concat(2, [inputs, features]) # Concatenate along the 3rd dimension