I use Metal
and CADisplayLink
to live filter a CIImage
and render it into a MTKView
.
// Starting display link
displayLink = CADisplayLink(target: self, selector: #selector(applyAnimatedFilter))
displayLink.preferredFramesPerSecond = 30
displayLink.add(to: .current, forMode: .default)
@objc func applyAnimatedFilter() {
...
metalView.image = filter.applyFilter(image: ciImage)
}
According to the memory monitor in Xcode, memory usage is stable on iPhone X and never goes above 100mb, on devices like iPhone 6 or iPhone 6s the memory usage keeps growing until eventually the system kills the app.
I've checked for memory leaks using Instruments
, but no leaks were reported. Running the app through Allocations also don't show any problems and the app won't get shut down by the system. I also find it interesting that on newer devices the memory usage is stable but on older it just keeps growing and growing.
The filter's complexity don't matter as I tried even most simple filters and the issue persists. Here is an example from my metal file:
extern "C" { namespace coreimage {
float4 applyColorFilter(sample_t s, float red, float green, float blue) {
float4 newPixel = s.rgba;
newPixel[0] = newPixel[0] + red;
newPixel[1] = newPixel[1] + green;
newPixel[2] = newPixel[2] + blue;
return newPixel;
}
}
I wonder what can cause the issue on older devices and in which direction I should look up to.
Update 1: here are two 1 minute graphs, one from Xcode
and one from Allocations
both using the same filter. Allocations
graph is stable while Xcode
graph is always growing:
Update 2: Attaching a screenshot of Allocations List sorted by size, the app was running for 16 minutes, applying the filter non stop:
Update 3: A bit more info on what is happening in applyAnimatedFilter()
:
I render a filtered image into a metalView
which is a MTKView
. I receive the filtered image from filter.applyFilter(image: ciImage)
, where in Filter
class happens next:
func applyFilter(image: ciImage) -> CIImage {
...
var colorMix = ColorMix()
return colorMix.use(image: ciImage, time: filterTime)
}
where filterTime
is just a Double variable. And finally, here is the whole ColorMix class:
import UIKit
class ColorMix: CIFilter {
private let kernel: CIKernel
@objc dynamic var inputImage: CIImage?
@objc dynamic var inputTime: CGFloat = 0
override init() {
let url = Bundle.main.url(forResource: "default", withExtension: "metallib")!
let data = try! Data(contentsOf: url)
kernel = try! CIKernel(functionName: "colorMix", fromMetalLibraryData: data)
super.init()
}
required init?(coder aDecoder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
func outputImage() -> CIImage? {
guard let inputImage = inputImage else {return nil}
return kernel.apply(extent: inputImage.extent, roiCallback: {
(index, rect) in
return rect.insetBy(dx: -1, dy: -1)
}, arguments: [inputImage, CIVector(x: inputImage.extent.width, y: inputImage.extent.height), inputTime])
}
func use(image: CIImage, time: Double) -> CIImage {
var resultImage = image
// 1. Apply filter
let filter = ColorMix()
filter.setValue(resultImage, forKey: "inputImage")
filter.setValue(NSNumber(floatLiteral: time), forKey: "inputTime")
resultImage = filter.outputImage()!
return resultImage
}
}
Here are a few observations, but I'm not sure if one of them actually causes the memory usage you're seeing:
applyFilter
you are creating a new ColorMix
filter every frame . Additionally, inside the instance method use(image:, time:)
you are creating another one on every call. That's a lot of overhead, especially since the filter loads it's kernel every time on init
. It would be advisable to just create a single ColorMix
filter during setup and just update its inputImage
and inputTime
on every frame. outputImage
is not a func
, but a var
that you override from the CIFilter
super class:
override var outputImage: CIImage? { /* your code here */ }
Is your colorMix
kernel performing any kind of convolution? If not, it could be a CIColorKernel
instead.
.size()
on the input sampler
. This is a bug in Xcode's diagnostic features (Metal validation and/or GPU frame capture). If you turn those off, the memory usage should be similar to when running outside of Xcode.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.