rgb_to_grayscale with tensorflow 2.3, getting “dimensions must be equal error”

Question

I'm a tensorflow noob, sorry. I am following this tutorial: https://www.tensorflow.org/tutorials/images/segmentation

I want to apply some preprocessing to the images in the dataset, and rgb_to_grayscale is failing with the error below. My median filter works, just not rbg_to_grayscale.

I would really appreciate any advice you have.

@tf.function
def load_image_train(datapoint):
  input_image = tf.image.resize(datapoint['image'], (128, 128))
  input_mask = tf.image.resize(datapoint['segmentation_mask'], (128, 128))  

  print(type(input_image))
  print(input_image.shape)

  input_image = tf.image.rgb_to_grayscale(input_image)
  input_mask = tf.image.rgb_to_grayscale(input_mask)

  input_image = tfa.image.median_filter2d(input_image)
  input_mask = tfa.image.median_filter2d(input_mask)

print output:

<class 'tensorflow.python.framework.ops.Tensor'>
(128, 128, 3)

Errror received:

    tutorial.py:36 load_image_train  *
        input_mask = tf.image.rgb_to_grayscale(input_mask)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper  **
        return target(*args, **kwargs)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/ops/image_ops_impl.py:2136 rgb_to_grayscale
        gray_float = math_ops.tensordot(flt_image, rgb_weights, [-1, -1])
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:4519 tensordot
        ab_matmul = matmul(a_reshape, b_reshape)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3255 matmul
        a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py:5642 mat_mul
        name=name)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:744 _apply_op_helper
        attrs=attr_protos, op_def=op_def)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py:593 _create_op_internal
        compute_device)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:3485 _create_op_internal
        op_def=op_def)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1975 __init__
        control_input_ops, op_def)
    /home/dmattie/environments/imgproc/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1815 _create_c_op
        raise ValueError(str(e))

    ValueError: Dimensions must be equal, but are 1 and 3 for '{{node rgb_to_grayscale_1/Tensordot/MatMul}} = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false](rgb_to_grayscale_1/Tensordot/Reshape, rgb_to_grayscale_1/Tensordot/Reshape_1)' with input shapes: [16384,1], [3,1].

Answer 1

The shape of input_mask is (128,128,1) so that when it gets flattened there is only one value in the last axis. This makes it incompatible with tf.image.rgb_to_grayscale which requires three RGB values in the last axis. You should not be interpreting the mask as an image, but if you really want to be able to apply the greyscale to the mask values (I don't see any reason to do this), you could broadcast it:

input_mask = tf.broadcast_to(input_mask, (128,128,3))