How can we get H.264 encoded video stream from iPhone Camera?

Question

I am using following to get video sample buffer:

- (void) writeSampleBufferStream:(CMSampleBufferRef)sampleBuffer ofType:(NSString *)mediaType

Now my question is that how can I get h.264 encoded NSData from above sampleBuffer . Please suggest.

Answer 1

You can only get raw video images in either BGRA or YUV color formats from AVFoundation. However, when you write those frames to an mp4 via AVAssetWriter, they will be encoded using H264 encoding.

A good example with code on how to do that is RosyWriter

Note that after each AVAssetWriter write, you will know that one complete H264 NAL was written to a mp4. You could write code that reads a complete H264 NAL after each write by AVAssetWriter, which is going to give you access to an H264 encoded frame. It might take a bit to get it right with decent speed, but it is doable( I did it successfully).

By the way, in order to successfully decode these encoded video frames, you will need H264 SPS and PPS information which is located in a different place in the mp4 file. In my case, I actually create couple of test mp4 files, and then manually extracted those out. Since those don't change, unless you change the H264 encoded specs, you can use them in your code.

Check my post to SPS values for H 264 stream in iPhone to see some of the SPS/PPS I used in my code.

Just a final note, in my case I had to stream h264 encoded frames to another endpoint for decoding/viewing; so my code had to do this fast. In my case, it was relatively fast; but eventually I switched to VP8 for encoding/decoding just because it was way faster because everything was done in memory without file reading/writing.

Good luck, and hopefully this info helps.

Answer 2

Update for 2017:

You can do streaming Video and Audio now by using the VideoToolbox API. Read the documentation here: VTCompressionSession

Original answer (from 2013):

Short: You can't, the sample buffer you receive is uncompressed.

Methods to get hardware accelerated h264 compression:

• AVAssetWriter
• AVCaptureMovieFileOutput

As you can see both write to a file, writing to a pipe does not work as the encoder updates header information after a frame or GOP has been fully written. So you better don't touch the file while the encoder writes to it as it does randomly rewrite header information. Without this header information the video file will not be playable (it updates the size field, so the first header written says the file is 0 bytes). Directly writing to a memory area is not supported currently. But you can open the encoded video-file and demux the stream to get to the h264 data ( after the encoder has closed the file of course)

Answer 3

Use VideoToolbox API. refer: https://developer.apple.com/videos/play/wwdc2014/513/

import Foundation
import AVFoundation
import VideoToolbox

public class LiveStreamSession {
    
    let compressionSession: VTCompressionSession
        
    var index = -1
    
    var lastInputPTS = CMTime.zero
    
    public init?(width: Int32, height: Int32){
        var compressionSessionOrNil: VTCompressionSession? = nil
        let status = VTCompressionSessionCreate(allocator: kCFAllocatorDefault,
                                                width: width,
                                                height: height,
                                                codecType: kCMVideoCodecType_H264,
                                                encoderSpecification: nil, // let the video toolbox choose a encoder
                                                imageBufferAttributes: nil,
                                                compressedDataAllocator: kCFAllocatorDefault,
                                                outputCallback: nil,
                                                refcon: nil,
                                                compressionSessionOut: &compressionSessionOrNil)
        guard status == noErr,
            let compressionSession = compressionSessionOrNil else {
            return nil
        }
        VTSessionSetProperty(compressionSession, key: kVTCompressionPropertyKey_RealTime, value: kCFBooleanTrue);
        VTCompressionSessionPrepareToEncodeFrames(compressionSession)
        
        self.compressionSession = compressionSession
        
    }
    
    public func pushVideoBuffer(buffer: CMSampleBuffer) {
        // image buffer
        guard let imageBuffer = CMSampleBufferGetImageBuffer(buffer) else {
            assertionFailure()
            return
        }
        
        // pts
        let pts = CMSampleBufferGetPresentationTimeStamp(buffer)
        guard CMTIME_IS_VALID(pts) else {
            assertionFailure()
            return
        }
        
        // duration
        var duration = CMSampleBufferGetDuration(buffer);
        if CMTIME_IS_INVALID(duration) && CMTIME_IS_VALID(self.lastInputPTS) {
            duration = CMTimeSubtract(pts, self.lastInputPTS)
        }
                
        index += 1
        self.lastInputPTS = pts
        print("[\(Date())]: pushVideoBuffer \(index)")
        
        let currentIndex = index
        VTCompressionSessionEncodeFrame(compressionSession, imageBuffer: imageBuffer, presentationTimeStamp: pts, duration: duration, frameProperties: nil, infoFlagsOut: nil) {[weak self] status, encodeInfoFlags, sampleBuffer in
            print("[\(Date())]: compressed \(currentIndex)")
            if let sampleBuffer = sampleBuffer {
                self?.didEncodeFrameBuffer(buffer: sampleBuffer, id: currentIndex)
            }
        }
    }
    
    deinit {
        VTCompressionSessionInvalidate(compressionSession)
    }
    
    private func didEncodeFrameBuffer(buffer: CMSampleBuffer, id: Int) {
        guard let attachments = CMSampleBufferGetSampleAttachmentsArray(buffer, createIfNecessary: true)
               else {
            return
        }
        let dic = Unmanaged<CFDictionary>.fromOpaque(CFArrayGetValueAtIndex(attachments, 0)).takeUnretainedValue()
        let keyframe = !CFDictionaryContainsKey(dic, Unmanaged.passRetained(kCMSampleAttachmentKey_NotSync).toOpaque())
//        print("[\(Date())]: didEncodeFrameBuffer \(id) is I frame: \(keyframe)")
        if keyframe,
           let formatDescription = CMSampleBufferGetFormatDescription(buffer) {
            // https://www.slideshare.net/instinctools_EE_Labs/videostream-compression-in-ios
            var number = 0
            CMVideoFormatDescriptionGetH264ParameterSetAtIndex(formatDescription, parameterSetIndex: 0, parameterSetPointerOut: nil, parameterSetSizeOut: nil, parameterSetCountOut: &number, nalUnitHeaderLengthOut: nil)
            // SPS and PPS and so on...
            let parameterSets = NSMutableData()
            for index in 0 ... number - 1 {
                var parameterSetPointer: UnsafePointer<UInt8>?
                var parameterSetLength = 0
                CMVideoFormatDescriptionGetH264ParameterSetAtIndex(formatDescription, parameterSetIndex: index, parameterSetPointerOut: &parameterSetPointer, parameterSetSizeOut: &parameterSetLength, parameterSetCountOut: nil, nalUnitHeaderLengthOut: nil)
//                parameterSets.append(startCode, length: startCodeLength)
                if let parameterSetPointer = parameterSetPointer {
                    parameterSets.append(parameterSetPointer, length: parameterSetLength)
                }
                
                //
                if index == 0 {
                    print("SPS is \(parameterSetPointer) with length \(parameterSetLength)")
                } else if index == 1 {
                    print("PPS is \(parameterSetPointer) with length \(parameterSetLength)")
                }
            }
            print("[\(Date())]: parameterSets \(parameterSets.length)")
        }
    }
}