ValueError: Dimensions must be equal, but are 4096 and 9 for 'mul'. Why no broadcasting here?

Question

I have a very simple example:

import tensorflow as tf
import pdb

number_features = tf.random_uniform((4096,22))

probs = number_features
probs_L = probs[:,:3]
probs_S1 = probs[:,3:12]
probs_S2 = probs[:,12:22]

confidence_no_digits = probs_L[:,0]
confidence_single_digit = probs_L[:,1] * probs_S1

with tf.Session() as sess:
    result = sess.run([confidence_single_digit])

However this gives:

ValueError: Dimensions must be equal, but are 4096 and 9 for 'mul' (op: 'Mul') with input shapes: [4096], [4096,9].

Why can I not multiply the vector of size [4096] and a matrix of size [4096,9] element-wise. Why does broadcasting not work here?

Answer 1

Broadcasting in tensorflow follows the same patterns as NumPy broadcasting . When operating on two arrays, it compares their shapes element-wise, starting with the last dimension, and works its way to the first dimension. Two dimensions are compatible when:

• they are equal, or
• one of them is 1, or
• one dimension is missing

In this case, starting from the last dimensions, the dimensions 4096 (the last dimension of the first array) and 9 (the last dimension of the second array) are not compatible according to the above rules, therefore giving you an error.

In order to fix it to get your desired broadcasting effect, you can transform the first array to have a compatible shape:

confidence_single_digit = tf.expand_dims(probs_L[:,1],1) * probs_S1

So that the shapes are (4096, 1) and (4096, 9) respectively.

Answer 2

If I'm not mistaken, the * symbol means element-wise multiplication, while you want matrix multiplication. You should rather use TF's matrix multiplication function matmul .

Try:

confidence_single_digit = tf.matmul(probs_L[:,1], probs_S1)

Update: In case you want element-wise multiplication, then use the normal multiplication function. This can be seen in this question.

Try:

confidence_single_digit = tf.multiply(probs_L[:,1], probs_S1)

Note: I have never used TensorFlow before. This could be a starting point of where to look for the bug.

Answer 3

Do you get the desired result with this ?

confidence_single_digit = tf.expand_dims(probs_L[:,1],1) * probs_S1

Now the shapes are these.

<bound method Tensor.get_shape of <tf.Tensor 'ExpandDims_1:0' 
shape=(4096, 1) dtype=float32>>

<bound method Tensor.get_shape of <tf.Tensor 'strided_slice_2:0' 
shape=(4096, 9) dtype=float32>>