Rewriting an Android OpenGL filter to Metal (for CIFilter)

Question

There are dozens of image filters written for the Android version of our app in GLSL (ES). As of iOS 12, OpenGL is deprecated, and CIFilter kernels have to be written in Metal.

I had some previous background in OpenGL, however writing CIFilter kernels in Metal is new to me.

Here is one of the filters. Could you help me in translating it to Metal as a CIFilter kernel? That would provide a good example for me so I could translate others.

#extension GL_OES_EGL_image_external : require
precision mediump float;
varying vec2 vTextureCoord;
uniform samplerExternalOES sTexture;
uniform float texelWidth;
uniform float texelHeight;
uniform float intensivity;
void main() {
    float SIZE = 1.25 + (intensivity / 100.0)*2.0;
    vec4 color;
    float min = 1.0;
    float max = 0.0;
    float val = 0.0;
    for (float x = -SIZE; x < SIZE; x++) {
        for (float y = -SIZE; y < SIZE; y++) {
            color = texture2D(sTexture, vTextureCoord + vec2(x * texelWidth, y * texelHeight));
            val = (color.r + color.g + color.b) / 3.;
            if (val > max) { max = val; } else if (val < min) { min = val; }
        }
    }
    float range = 5. * (max - min);
    gl_FragColor = vec4(pow(1. - range, SIZE * 1.5));
    gl_FragColor = vec4((gl_FragColor.r + gl_FragColor.g + gl_FragColor.b) / 3. > 0.75 ? vec3(1.) : gl_FragColor.rgb, 1.);
}

Answer 1

Here's the Metal source for a kernel that attempts to replicate your described filter:

#include <metal_stdlib>
#include <CoreImage/CoreImage.h>

using namespace metal;

extern "C" {
namespace coreimage {

float4 sketch(sampler src, float texelWidth, float texelHeight, float intensity40) {
    float size = 1.25f + (intensity40 / 100.0f) * 2.0f;

    float minVal = 1.0f;
    float maxVal = 0.0f;
    for (float x = -size; x < size; ++x) {
        for (float y = -size; y < size; ++y) {
            float4 color = src.sample(src.coord() + float2(x * texelWidth, y * texelHeight));
            float val = (color.r + color.g + color.b) / 3.0f;
            if (val > maxVal) {
                maxVal = val;
            } else if (val < minVal) {
                minVal = val;
            }
        }
    }

    float range = 5.0f * (maxVal - minVal);

    float4 outColor(pow(1.0f - range, size * 1.5f));
    outColor = float4((outColor.r + outColor.g + outColor.b) / 3.0f > 0.75f ? float3(1.0f) : outColor.rgb, 1.0f);
    return outColor;
}

}
}

I assume you're already familiar with the basics of how to correctly build Metal shaders into a library that can be loaded by Core Image.

You can instantiate your kernel at runtime by loading the default Metal library and requesting the "sketch" function (the name is arbitrary, so long as it matches the kernel source):

NSURL *libraryURL = [NSBundle.mainBundle URLForResource:@"default" withExtension:@"metallib"];
NSData *libraryData = [NSData dataWithContentsOfURL:libraryURL];

NSError *error;
CIKernel *kernel = [CIKernel kernelWithFunctionName:@"sketch" fromMetalLibraryData:libraryData error:&error];

You can then apply this kernel to an image by wrapping it in your own CIFilter subclass, or just invoke it directly:

CIImage *outputImage = [kernel applyWithExtent:CGRectMake(0, 0, width, height)
                                   roiCallback:^CGRect(int index, CGRect destRect)
                        { return destRect; }
                                     arguments:@[inputImage, @(1.0f/width), @(1.0f/height), @(60.0f)]];

I've tried to select sensible defaults for each of the arguments (the first of which should be an instance of CIImage ), but of course these can be adjusted to taste.