使用 AV 播放器访问单个帧

Question

In a recent project, I have to access all frames of my video individually using AV Foundation. Also, if possible to acess them randomly (like an array)

I tried to research the question but I didn't get anything useful.

Note: Is there any useful documentation to get familiar with the AV Foundation ?

Answer 1

You can enumerate the frames of your video serially using AVAssetReader , like this:

let asset = AVAsset(URL: inputUrl)
let reader = try! AVAssetReader(asset: asset)

let videoTrack = asset.tracksWithMediaType(AVMediaTypeVideo)[0]

// read video frames as BGRA
let trackReaderOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings:[String(kCVPixelBufferPixelFormatTypeKey): NSNumber(unsignedInt: kCVPixelFormatType_32BGRA)])

reader.addOutput(trackReaderOutput)
reader.startReading()

while let sampleBuffer = trackReaderOutput.copyNextSampleBuffer() {
    print("sample at time \(CMSampleBufferGetPresentationTimeStamp(sampleBuffer))")
    if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
        // process each CVPixelBufferRef here
        // see CVPixelBufferGetWidth, CVPixelBufferLockBaseAddress, CVPixelBufferGetBaseAddress, etc
    }
}

Random access is more complicated. You could use an AVPlayer + AVPlayerItemVideoOutput to get frames from any time t , using copyPixelBufferForItemTime , as described in this answer , but the subtlety lies in how you choose that t .

If you want to sample the video at uniform intervals, then that's easy, but if you want to land on the same frames/presentation time stamps that the serial AVAssetReader code sees, then you will probably have to preprocess the file with AVAssetReader , to build a frame number -> presentation timestamp map. This can be fast if you skip decoding by using nil output settings in AVAssetReaderTrackOutput .

Answer 2

If you need to get the current frame while playing the video you could add an observer for the player and could get the current frame like this:

var player: AVPlayer?
var playerController = AVPlayerViewController()
var videoFPS: Int = 0
var currentFrame: Int = 0
var totalFrames: Int?

func configureVideo() {
    guard let videoURL = "" else { return }
    player = AVPlayer(url: videoURL)
    playerController.player = player
    guard player?.currentItem?.asset != nil else {
        return
    }
    let asset = self.player?.currentItem?.asset
    let tracks = asset!.tracks(withMediaType: .video)
    let fps = tracks.first?.nominalFrameRate
    
    self.videoFPS = lround(Double(fps!))
    self.getVideoData()
}
  
func getVideoData() {
    self.player?.addPeriodicTimeObserver(forInterval: CMTimeMake(value: 1,timescale: Int32(self.videoFPS)), queue: DispatchQueue.main) {[weak self] (progressTime) in

        if let duration = self!.player?.currentItem?.duration {

            let durationSeconds = CMTimeGetSeconds(duration)
            let seconds = CMTimeGetSeconds(progressTime)
            if self!.totalFrames == nil {
                self!.totalFrames = lround(Double(self!.videoFPS) * durationSeconds)
            }

            DispatchQueue.main.async {
                if self!.totalFrames != self!.currentFrame {
                    self!.currentFrame = lround(seconds*Double(self!.videoFPS))
                } else {
                    print("Video has ended!!!!!!!!")
                }
                print(self!.currentFrame)
            }
        }
    }
}

After getting the current frame number, you could use an AVAssetImageGenerator to retrieve the specific image.