使用Metal的旧设备的内存使用率不断上升

Question

I use Metal and CADisplayLink to live filter a CIImage and render it into a MTKView .

// Starting display link 
displayLink = CADisplayLink(target: self, selector: #selector(applyAnimatedFilter))
displayLink.preferredFramesPerSecond = 30
displayLink.add(to: .current, forMode: .default)

@objc func applyAnimatedFilter() {
    ...
    metalView.image = filter.applyFilter(image: ciImage)
}

According to the memory monitor in Xcode, memory usage is stable on iPhone X and never goes above 100mb, on devices like iPhone 6 or iPhone 6s the memory usage keeps growing until eventually the system kills the app.

I've checked for memory leaks using Instruments , but no leaks were reported. Running the app through Allocations also don't show any problems and the app won't get shut down by the system. I also find it interesting that on newer devices the memory usage is stable but on older it just keeps growing and growing.

The filter's complexity don't matter as I tried even most simple filters and the issue persists. Here is an example from my metal file:

extern "C" { namespace coreimage {

    float4 applyColorFilter(sample_t s, float red, float green, float blue) {

        float4 newPixel = s.rgba;
        newPixel[0] = newPixel[0] + red;
        newPixel[1] = newPixel[1] + green;
        newPixel[2] = newPixel[2] + blue;

        return newPixel;
    }
}

I wonder what can cause the issue on older devices and in which direction I should look up to.

Update 1: here are two 1 minute graphs, one from Xcode and one from Allocations both using the same filter. Allocations graph is stable while Xcode graph is always growing:

Update 2: Attaching a screenshot of Allocations List sorted by size, the app was running for 16 minutes, applying the filter non stop:

Update 3: A bit more info on what is happening in applyAnimatedFilter() :

I render a filtered image into a metalView which is a MTKView . I receive the filtered image from filter.applyFilter(image: ciImage) , where in Filter class happens next:

 func applyFilter(image: ciImage) -> CIImage {
    ...
    var colorMix = ColorMix()
    return colorMix.use(image: ciImage, time: filterTime)
 }

where filterTime is just a Double variable. And finally, here is the whole ColorMix class:

import UIKit

class ColorMix: CIFilter {

    private let kernel: CIKernel

    @objc dynamic var inputImage: CIImage?
    @objc dynamic var inputTime: CGFloat = 0

    override init() {

        let url = Bundle.main.url(forResource: "default", withExtension: "metallib")!
        let data = try! Data(contentsOf: url)
        kernel = try! CIKernel(functionName: "colorMix", fromMetalLibraryData: data)
        super.init()
    }

    required init?(coder aDecoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    func outputImage() -> CIImage? {

        guard let inputImage = inputImage else {return nil}

        return kernel.apply(extent: inputImage.extent, roiCallback: {
            (index, rect) in
            return rect.insetBy(dx: -1, dy: -1)
        }, arguments: [inputImage, CIVector(x: inputImage.extent.width, y: inputImage.extent.height), inputTime])
    }

    func use(image: CIImage, time: Double) -> CIImage {

        var resultImage = image

        // 1. Apply filter
        let filter = ColorMix()
        filter.setValue(resultImage, forKey: "inputImage")
        filter.setValue(NSNumber(floatLiteral: time), forKey: "inputTime")

        resultImage = filter.outputImage()!

        return resultImage
    }

}

Answer 1

Here are a few observations, but I'm not sure if one of them actually causes the memory usage you're seeing:

• In applyFilter you are creating a new ColorMix filter every frame . Additionally, inside the instance method use(image:, time:) you are creating another one on every call. That's a lot of overhead, especially since the filter loads it's kernel every time on init . It would be advisable to just create a single ColorMix filter during setup and just update its inputImage and inputTime on every frame.

• outputImage is not a func , but a var that you override from the CIFilter super class:

  override var outputImage: CIImage? { /* your code here */ }

• Is your colorMix kernel performing any kind of convolution? If not, it could be a CIColorKernel instead.

• If you need the size of the input inside your kernel, you don't need to pass it as extra argument. You can just call .size() on the input sampler .

Answer 2

This is a bug in Xcode's diagnostic features (Metal validation and/or GPU frame capture). If you turn those off, the memory usage should be similar to when running outside of Xcode.