How to generate CNN model with changing frames in tensorflow?

Question

I want to build a conditional CNN model in Tensorflow, but I met some trouble with it.

Suppose there is a matrix named with shape [64, ?, 50, 1] and another matrix named with shape [64, ?, 1, 130]. [64，？，50,1]的特征的矩阵和另一个名为 [64，？，1,11]的矩阵。

The fisrt dim is the batch size, and the second dim in means the frame number ( ). ）。 The third one is the feature dimention, and the last dim is channel num.

I want to concat the two matrix, which means we can get a matrix with shape [64, ?, 50, 131], and do in tensorflow. ]的矩阵，并在中进行conv2d。

But the problem is that I cannot concat these two matrices because of the difference in the third dim. Then I did as follows:

    x_shapes = x.get_shape()
    y_shapes = y.get_shape()
    return tf.concat(3, [x, y*tf.ones([x_shapes[0], x_shapes[1], x_shapes[2], y_shapes[3]])])

But it doesn't work because the second dim is not known.

I wonder is there any way to solve this problem?

Thanks

Answer 1

Although it is not clear from your question, it looks like you want to broadcast the second tensor with shape [64, ?, 1, 130] on the 3rd dimension, the one you call feature dimension, before you concatenate. Note that tf.concat needs all the dimensions to match, except the one along which you are concatenating. From the documentation of tf.concat :

The number of dimensions of the input tensors must match, and all dimensions except concat_dim must be equal.

To do the broadcast along the feature dimension, it is much cheaper to use tf.tile instead of multiplying with a all ones tensor like you are doing. Here is how you would use tf.tile :

tf.concat(3, [x, tf.tile(y, [1, 1, x_shapes.as_list()[2], 1])])

In your case, since the 3rd dimension of y is known statically, the above code works. If that's not the case, you can form the second argument to tf.tile as follows :

tf.tile(y, tf.concat(0, [[1], [1], tf.shape(x)[2:3], [1]]))