如何在Swift中从本地视频文件中获取帧？

Question

I need to get the frames from a local video file so i can process them before the video is played. I already tried using AVAssetReader and VideoOutput.

[EDIT] Here is the code i used from Accesing Individual Frames using AV Player

let asset = AVAsset(URL: inputUrl)
let reader = try! AVAssetReader(asset: asset)

let videoTrack = asset.tracksWithMediaType(AVMediaTypeVideo)[0]

// read video frames as BGRA
let trackReaderOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings:[String(kCVPixelBufferPixelFormatTypeKey): NSNumber(unsignedInt: kCVPixelFormatType_32BGRA)])

reader.addOutput(trackReaderOutput)
reader.startReading()

while let sampleBuffer = trackReaderOutput.copyNextSampleBuffer() {
    print("sample at time \(CMSampleBufferGetPresentationTimeStamp(sampleBuffer))")
    if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
        // process each CVPixelBufferRef here
        // see CVPixelBufferGetWidth, CVPixelBufferLockBaseAddress, CVPixelBufferGetBaseAddress, etc
    }
}

Answer 1

I believe AVAssetReader should work. What did you try? Have you seen this sample code from Apple? https://developer.apple.com/library/content/samplecode/ReaderWriter/Introduction/Intro.html

Answer 2

You can have a look at VideoToolbox : https://developer.apple.com/documentation/videotoolbox

But beware: this is close to the hardware decompressor and sparsely documented terrain.

Answer 3

Depending on what processing you want to do, OpenCV may be a an option - in particular if you are detecting or tracking objets in your frames. If your needs are simpler, then the effort to use OpenCV with swift may be a little too much - see below.

You can open a video, read it frame by frame, do your work on the frames and then display then - bearing in mind the need to be efficient to avoid delaying the display.

The basic code structure is quite simple - this is a python example but the same principles apply across supported languages

import numpy as np
import cv2

cap = cv2.VideoCapture('vtest.avi')

while(cap.isOpened()):
    ret, frame = cap.read()

    //Do whatever work you want on the frame here - in this example
    //from the tutorial the image is being converted from one colour 
    //space to another
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    //This displays the resulting frame 
    cv2.imshow('frame',gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

More info here: http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_gui/py_video_display/py_video_display.html

The one caveat is that using OpenCV with swift requires some additional effort - this is a good example, but it evolves constantly so it is worth searching for if you decide to go this way: https://medium.com/@yiweini/opencv-with-swift-step-by-step-c3cc1d1ee5f1

Answer 4

I found out what the problem was! It was with my implementation. The code i posted is correct. Thank you all