如何将多个 CGImage 合并成一个更大的 CGImage？

Question

I have created an app that applies a neural network (convnet) to an input image followed by a post-processing. This convnet is basically a filter that inputs an image (plus one parameter) and outputs an image of similar size. Since the convnet cannot process a large image in one pass due to memory issue, the image must be split in tiles which are then sticked (or unified) together after the model is applied. My problem is about image manipulation. It will be more clear after I present you what is done in details:

• Input image is a UIImage
• Split the input image into as a list of UIImage called listInput
• Create an empty listOutput
• For each tile in listInput :
  • Convert the UIImage into a CGImage
  • Convert the CGImage into a CVPixelBuffer
  • Apply a CoreML model to the CVPixelBuffer which returns a CVPixelBuffer of the same size
  • Convert the CVPixelBuffer into a CIImage
  • Convert the CIImage into a CGImage
  • Convert the CGImage into a UIImage
  • Append the UIImage into listOutput
• Unify all the tiles in listOutput into an output UIImage
• Fuse input and output UIImage (post-processing):
  • Convert input UIImage into CGImage then to CIImage
  • Convert output UIImage into CGImage then to CIImage
  • Fuse the 2 CIImage using a CIFilter
  • Convert the resulting CIImage into a CGImage
  • Convert the CGImage into a UIImage

I can post the code corresponding to any of the part listed above if needed.

The general problem I have is all the conversions between UIImage to CGImage to CIImage and conversly. I'm trying to get rid of UIImage completely (except for loading the image). I want indeed to manipulate CGImage from the start until the end. This will already simplify the code.

I've modified my code to manipulate list of CGImage instead of list of UIImage. The cropping part is in fact simpler with CGImage than with UIImage. But I cannot figure out the other way around: unify CGImage together into a bigger image. This is my specific problem. Bellow is the function I've created to unify the UIImage.

func unifyTiles(listTiles: [UIImage], listRect: [CGRect]) -> UIImage? {

    guard let input = input else {
        return nil
    }


    let outputSize = CGSize(width : Int(input.size.width), height: Int(input.size.height))

    UIGraphicsBeginImageContextWithOptions(outputSize, true, 1.0)

    for i in 0..<listTiles.count {
        listTiles[i].draw(at: listRect[i].origin)
    }

    guard let output = UIGraphicsGetImageFromCurrentImageContext() else {
        return nil
    }
    UIGraphicsEndImageContext()

    return output
}

So my question is:

• Is it possible to do the same just manipulating CGImage?
• Is it even a good idea?

Some notes:

• The post-procesing must be separated from the previous part because the user wants to modify the post-processing parameters without reapplying the convnet. The application of the convnet is indeed very long and can take up to a minute to compute on large images while the post-processing is near real-time.

• In the post-processing part, it was suggested to me to convert directly UIImage <-> CIImage without going through CGImage. For some reason that I don't know, this doesn't work as far as I remember.

• I'm aware that using Vision I could feed directly a CGImage into the network instead of a CVPixelBuffer, but I don't know if Vision can output a CGImage as well. This will be investigated soon hopefully.

Thanks for any information you could give me.

Answer 1

UIImage to CIImage

This step:

UIImage into CGImage then to CIImage

is overblown, as CIImage has an init(image:) initializer that goes directly from UIImage to CIImage.

Cropping

You seem to think that cropping a CGImage is easier than cropping a UIImage, but it isn't. To crop a UIImage, just draw it into a smaller graphics context, offset so as to place the desired point at the top left of the crop.

Graphics Contexts

You can only draw in a graphics context, and that's going to be an image context whether you like it or not. There's no such thing as drawing into a "CGImage context" if that's what you're thinking. You can draw a CGImage directly into an image context, but the results are usually disastrous (as I shall explain in the next paragraph).

Final thoughts

In general I would like to set your mind at rest about UIImage. A UIImage is a good thing. It is (usually) a very lightweight wrapper around a CGImage. The CGImage is the bitmap data; the wrapper adds information like scale and orientation. Losing that information can cause your drawing to come out very badly; trying to draw a CGImage can cause the drawing to be flipped and incorrectly scaled. Don't do it! Use UIImage and be happy.