Applying a CIFilter to a Video File and Saving it

Question

Is there any fast, lightweight-as-possible way to apply a CIFilter to a video? Before it's mentioned, I have looked at GPUImage - it looks like very powerful magic code, but it's really overkill for what I'm trying to do.

Essentially, I would like to

1. Take a video file, say stored at /tmp/myVideoFile.mp4
2. Apply a CIFilter to this video file
3. Save the video file to a different (or the same) location, say /tmp/anotherVideoFile.mp4

I've been able to apply a CIFilter to a video that's playing extremely easily and quickly using AVPlayerItemVideoOutput

let player = AVPlayer(playerItem: AVPlayerItem(asset: video))
let output = AVPlayerItemVideoOutput(pixelBufferAttributes: nil)
player.currentItem?.addOutput(self.output)
player.play()

let displayLink = CADisplayLink(target: self, selector: #selector(self.displayLinkDidRefresh(_:)))
displayLink.addToRunLoop(NSRunLoop.mainRunLoop(), forMode: NSRunLoopCommonModes)

func displayLinkDidRefresh(link: CADisplayLink){
    let itemTime = output.itemTimeForHostTime(CACurrentMediaTime())
    if output.hasNewPixelBufferForItemTime(itemTime){
        if let pixelBuffer = output.copyPixelBufferForItemTime(itemTime, itemTimeForDisplay: nil){
            let image = CIImage(CVPixelBuffer: pixelBuffer)
            // apply filters to image
            // display image
        }
    }
}

This works great, but I've been having a lot just the tiniest bit of trouble finding out how to apply a filter to an already saved video file. There is the option of basically just doing what I did above, using an AVPlayer , playing the video, and getting the pixel buffer from every frame as it is played but this won't work for video processing in the background. I don't think users would appreciate having to wait as long as their video is for the filter to be applied.

In way over-simplified code, I'm looking for something like this:

var newVideo = AVMutableAsset() // We'll just pretend like this is a thing

var originalVideo = AVAsset(url: NSURL(urlString: "/example/location.mp4"))
originalVideo.getAllFrames(){(pixelBuffer: CVPixelBuffer) -> Void in
    let image = CIImage(CVPixelBuffer: pixelBuffer)
        .imageByApplyingFilter("Filter", withInputParameters: [:])

    newVideo.addFrame(image)
}

newVideo.exportTo(url: NSURL(urlString: "/this/isAnother/example.mp4"))

Is there any way fast (again, not involving GPUImage, and ideally working in iOS 7) way to apply a filter to a video file and then save it? For example this would take a saved video, load it into an AVAsset , apply a CIFilter , and then save the new video to a different location.

Answer 1

In iOS 9 / OS X 10.11 / tvOS, there's a convenience method for applying CIFilter s to video. It works on an AVVideoComposition , so you can use it both for playback and for file-to-file import/export. See AVVideoComposition.init(asset:applyingCIFiltersWithHandler:) for the method docs.

There's an example in Apple's Core Image Programming Guide , too:

let filter = CIFilter(name: "CIGaussianBlur")!
let composition = AVVideoComposition(asset: asset, applyingCIFiltersWithHandler: { request in

    // Clamp to avoid blurring transparent pixels at the image edges
    let source = request.sourceImage.clampingToExtent()
    filter.setValue(source, forKey: kCIInputImageKey)

    // Vary filter parameters based on video timing
    let seconds = CMTimeGetSeconds(request.compositionTime)
    filter.setValue(seconds * 10.0, forKey: kCIInputRadiusKey)

    // Crop the blurred output to the bounds of the original image
    let output = filter.outputImage!.cropping(to: request.sourceImage.extent)

    // Provide the filter output to the composition
    request.finish(with: output, context: nil)
})

That part sets up the composition. After you've done that, you can either play it by assigning it to an AVPlayer or write it to a file with AVAssetExportSession . Since you're after the latter, here's an example of that:

let export = AVAssetExportSession(asset: asset, presetName: AVAssetExportPreset1920x1200)
export.outputFileType = AVFileTypeQuickTimeMovie
export.outputURL = outURL
export.videoComposition = composition

export.exportAsynchronouslyWithCompletionHandler(/*...*/)

There's a bit more about this in the WWDC15 session on Core Image, starting around 20 minutes in .

---

If you want a solution that works on earlier OS, it's a bit more complicated.

^{Aside: Think about how far back you really need to support. As of August 15, 2016 , 87% of devices are on iOS 9.0 or later, and 97% are on iOS 8.0 or later. Going to a lot of effort to support a small slice of your potential customer base—and it'll get even smaller by the time you get your project done and ready to deploy—might not be worth the cost.}

There are a couple of ways to go at this. Either way, you'll be getting CVPixelBuffer s representing source frames, creating CIImage s from them , applying filters, and rendering out new CVPixelBuffer s .

1. Use AVAssetReader and AVAssetWriter to read and write pixel buffers. There's examples for how to do this (the reading and writing part; you still need to do the filtering in between) in the Export chapter of Apple's AVFoundation Programming Guide.

2. Use AVVideoComposition with a custom compositor class. Your custom compositor is given AVAsynchronousVideoCompositionRequest objects that provide access to pixel buffers and a way for you to provide processed pixel buffers. Apple has a sample code project called AVCustomEdit that shows how to do this (again, just the getting and returning sample buffers part; you'd want to process with Core Image instead of using their GL renderers).

Of those two, the AVVideoComposition route is more flexible, because you can use a composition both for playback and export.