为具有灵活形状的 CoreML 2 model 指定输入/输出尺寸

Question

I managed to create a CoreML 2.0 model with flexible input/output shape sizes:

I can't figure out how to set the size in my Xcode project, however. If I set the input pixel buffer size 2048x2048, the output pixel buffer is still 1536x1536. If I set it to 768x768, the resulting pixel buffer is still 1536x1536 - but is blank outside the region of 768x768.

I examined the automatically generated Swift model class and don't see any clues there.

I can't find a single example anywhere showing how to use the "Flexibility" sizes.

In the WWDC 2018 Session 708 "What's New in Core ML", Part 1 it states:

This means that now you have to ship a single model. You don't have to have any redundant code. And if you need to switch between standard definition and high definition, you can do it much faster because we don't need to reload the model from scratch; we just need to resize it. You have two options to specify the flexibility of the model. You can define a range for its dimension, so you can define a minimal width and height and the maximum width and height. And then at inference pick any value in between. But there is also another way. You can enumerate all the shapes that you are going to use. For example, all different aspect ratios, all different resolutions, and this is better for performance. Core ML knows more about your use case earlier, so it can -- it has the opportunities of performing more optimizations.

They say "we just need to resize it". It so frustrating because they don't tell you how to just resize it! They also say "And then at inference pick any value in between" but offer no clue how to pick the value in between!

Here is how I added the flexible shape sizes:

import coremltools
from coremltools.models.neural_network import flexible_shape_utils
spec = coremltools.utils.load_spec('mymodel_fxedShape.mlmodel')
img_size_ranges = flexible_shape_utils.NeuralNetworkImageSizeRange()
img_size_ranges.add_height_range(640, 2048)
img_size_ranges.add_width_range(640, 2048)
flexible_shape_utils.update_image_size_range(spec, feature_name='inputImage', size_range=img_size_ranges)
flexible_shape_utils.update_image_size_range(spec, feature_name='outputImage', size_range=img_size_ranges)
coremltools.utils.save_spec(spec, 'myModel.mlmodel')

Here is the description of the model:

description {
  input {
    name: "inputImage"
    shortDescription: "Image to stylize"
    type {
      imageType {
        width: 1536
        height: 1536
        colorSpace: BGR
        imageSizeRange {
          widthRange {
            lowerBound: 640
            upperBound: 2048
          }
          heightRange {
            lowerBound: 640
            upperBound: 2048
          }
        }
      }
    }
  }
  output {
    name: "outputImage"
    shortDescription: "Stylized image"
    type {
      imageType {
        width: 1536
        height: 1536
        colorSpace: BGR
        imageSizeRange {
          widthRange {
            lowerBound: 640
            upperBound: 2048
          }
          heightRange {
            lowerBound: 640
            upperBound: 2048
          }
        }
      }
    }
  }
}

There are two layers using "outputShape":

layers {
    name: "SpatialFullConvolution_63"
    input: "Sequential_53"
    output: "SpatialFullConvolution_63_output"
    convolution {
      outputChannels: 16
      kernelChannels: 32
      nGroups: 1
      kernelSize: 3
      kernelSize: 3
      stride: 2
      stride: 2
      dilationFactor: 1
      dilationFactor: 1
      valid {
        paddingAmounts {
          borderAmounts {
          }
          borderAmounts {
          }
        }
      }
      isDeconvolution: true
      hasBias: true
      weights {
      }
      bias {
      }
      outputShape: 770
      outputShape: 770
    }
  }
  ...relu layer...
  layers {
    name: "SpatialFullConvolution_67"
    input: "ReLU_66"
    output: "SpatialFullConvolution_67_output"
    convolution {
      outputChannels: 8
      kernelChannels: 16
      nGroups: 1
      kernelSize: 3
      kernelSize: 3
      stride: 2
      stride: 2
      dilationFactor: 1
      dilationFactor: 1
      valid {
        paddingAmounts {
          borderAmounts {
          }
          borderAmounts {
          }
        }
      }
      isDeconvolution: true
      hasBias: true
      weights {
      }
      bias {
      }
      outputShape: 1538
      outputShape: 1538
    }
  }

I am now trying to figure out how to remove the outputShape from those two layers.

>>> layer = spec.neuralNetwork.layers[49]
>>> layer.convolution.outputShape
[1538L, 1538L]

I tried setting it to []:

layer.convolution.outputShape = []

To a Shape:

layer.convolution.outputShape = flexible_shape_utils.Shape(())

Whatever I try, I get the error:

TypeError: Can't set composite field

Do I have to create a new layer and then link it to the layer that is outputting to it and the layer it is outputting to?

Answer 1

The issue in this case was that there were layers present in the model that used a fixed shape for their outputShapes. For example:

>>> layer = spec.neuralNetwork.layers[49]
>>> layer.convolution.outputShape
[1538L, 1538L]

The model in question was indeed fully convolutional, so before converting to CoreML, it worked with any input and output shapes.

I was able to delete the fixed outputShape with this command:

layer = spec.neuralNetwork.layers[49]
del layer.convolution.outputShape[:]

After doing that, the model worked with flexible input and output shapes.

All credit for this answer goes to Matthijs Hollemans.