MLModel 与 MultiArray output 一起使用，但无法成功将 output 更改为图像

Question

I have converted a Keras model to a MLModel using coremltools 4.0 with limited success.

It works but only if I use an MLMultiArray for the output and covert to an image. Converting to an image takes magnitudes longer than inferencing; making it unusable.

If I try to change the MLModel spec to use images for output I get this error running prediction:

Failed to convert output Identity to image:

NSUnderlyingError=0x2809bad00 {Error Domain=com.apple.CoreML Code=0 "Invalid array shape ( 2048, 2048, 3 ) for converting to gray image"

Even though I have specified RGB for the output color:

output { name: "Identity" type { imageType { width: 2048 height: 2048 colorSpace: RGB } } }

If I use a MultiArray (that works) Xcode reports:

output: Float32 1 x 2048 x 2048 x 3 array

I suspect the issue is the first dimension, which is the batch number but no dimensions are shown, so I can't delete the batch dimension:

output { name: "Identity" type { multiArrayType { dataType: FLOAT32 } } }

I don't think I can just add an output shape to the Keras Conv2D output layer because it has multiple inbound nodes with different shapes. Here are the output shapes:

>>> print(outputLayer.get_output_shape_at(0))
(None, None, None, 3)
>>> print(outputLayer.get_output_shape_at(1))
(1, 512, 512, 3)
>>> print(outputLayer.get_output_shape_at(2))
(1, 2048, 2048, 3)

>>> print(outputLayer.output)
Tensor("SR/Identity:0", shape=(None, None, None, 3), dtype=float32)

I think coremltools is confusing the batch for the channels that is why it is attempting to create a grayscale image even if I specify RGB.

Any idea how to fix it?

I have the original Keras model, but I don't see how specify shapes without a batch dimension. Here is the beginning and ending of the Keras model layer description

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
LR_input (InputLayer)           [(None, None, None,  0                                            
__________________________________________________________________________________________________
Pre_blocks_conv (Conv2D)        multiple             896         LR_input[0][0]                   
__________________________________________________________________________________________________
F_1_1_1 (Conv2D)                multiple             9248        Pre_blocks_conv[0][0]            

...                             multiple
...                             multiple

SR (Conv2D)                     multiple             84          PixelShuffle[0][0]               
==================================================================================================

Answer 1

In Core ML the order of the dimensions is (channels, height, width) so it expects to see a 3 x 2048 x 2048 output instead of 2048 x 2048 x 3.

Note that you also need to make sure the output pixels are in the range [0, 255] instead of [0, 1] which is probably what your Keras model gives you.