在将图像提供给 CoreML 模型之前如何预处理图像？

Question

I created an image similarity model and used the reference data images to test it out. I tested the turicreate model, and I got back zero distances for reference data images, and the same came back when using this code with the coreml model:

image = tc.image_analysis.resize(reference_data[0]['image'], *reversed(model.input_image_shape))
image = PIL.Image.fromarray(image.pixel_data)
mlmodel.predict({'image':image})`

However, when using the model in iOS as a VNCoreMLModel, no reference image test came back with a zero distance, and most of them weren't even the shortest distance, ie reference image 0 had a shortest distance to reference id 78. Since the coreml model works in python, I figured it was a preprocessing issue, so I preprocessed the image myself before passing it to the CoreMLModel. Doing this gave me a consistent output of the reference ids matching the reference images for the shortest distance--yay. The distance still isn't zero, so I have attempted to do whatever I can think of to affect the image to get some difference, but I can't get it any closer to zero. Preprocessing code:

+ (CVPixelBufferRef)pixelBufferForImage:(UIImage *)image sideLength:(CGFloat)sideLength {
    UIGraphicsBeginImageContextWithOptions(CGSizeMake(sideLength, sideLength), YES, image.scale);
    [image drawInRect:CGRectMake(0, 0, sideLength, sideLength)];
    UIImage *resizedImage = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();

    CFStringRef keys[2] = {kCVPixelBufferCGImageCompatibilityKey, kCVPixelBufferCGBitmapContextCompatibilityKey};
    CFBooleanRef values[2] = {kCFBooleanTrue, kCFBooleanTrue};
    CFDictionaryRef attrs = CFDictionaryCreate(kCFAllocatorDefault, (const void **)keys, (const void **)values, 2, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks);
    CVPixelBufferRef buffer;
    int status = CVPixelBufferCreate(kCFAllocatorDefault, (int)(sideLength), (int)(sideLength), kCVPixelFormatType_32ARGB, attrs, &buffer);
    if (status != kCVReturnSuccess) {
        return nil;
    }

    CVPixelBufferLockBaseAddress(buffer, kCVPixelBufferLock_ReadOnly);
    void *data = CVPixelBufferGetBaseAddress(buffer);
    CGColorSpaceRef colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceSRGB);
    CGContextRef context = CGBitmapContextCreate(data, sideLength, sideLength, 8, CVPixelBufferGetBytesPerRow(buffer), colorSpace, kCGImageAlphaNoneSkipFirst);

    CGContextTranslateCTM(context, 0, sideLength);
    CGContextScaleCTM(context, 1.0, -1.0);

    UIGraphicsPushContext(context);
    [resizedImage drawInRect:CGRectMake(0, 0, sideLength, sideLength)];
    UIGraphicsPopContext();
    CVPixelBufferUnlockBaseAddress(buffer, kCVPixelBufferLock_ReadOnly);
    return buffer;
}

The mlmodel takes an RGB image with size: (224, 224)

What else can I do to the image to improve my results?

Answer 1

I was in the same boat as you. Since image preprocessing involves usage of blurring, conversion from RGB to gray and other steps. It would be easier to use Objective C++ wrapper. Below link gives good understanding about how it can be linked using header classes.

https://www.timpoulsen.com/2019/using-opencv-in-an-ios-app.html

Hope it helps!

Image Credits : https://medium.com/@borisohayon/ios-opencv-and-swift-1ee3e3a5735b