调用 broadcastPaused() 后 AVAssetWriter 异步视频和音频

Question

I am trying to record a video with sound from device screen using ReplayKit and RPBroadcastSampleHandler.

When i record it just using "start broadcast" and "stop" the result i get is great.

But if i try to pause recording (by using red status bar) i got problems. The result i got is video and audio with different length (audio is shorter but have all i need). On the recording i got video and audio that start being async after moment of tapping status bar(ios14). Audio goes good, but video freezing when status bar tapped and continue when modal window closed. As result i got video without audio in the end.

Here is my code:

1.All class fields i have:

class SampleHandler: RPBroadcastSampleHandler {
    
    private let videoService = VideoService()
    private let audioService = AudioService()
    private var isRecording = false
    
    private let lock = NSLock()
    private var finishCalled = false
    
    private var videoWriter: AVAssetWriter!
    private var videoWriterInput: AVAssetWriterInput!
    private var microphoneWriterInput: AVAssetWriterInput!
    private var sessionBeginAtSourceTime: CMTime!

2.Some configure on start capturing:

override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
    guard !isRecording else { return }
    isRecording = true
    BroadcastData.clear()
    BroadcastData.startVideoDate = Date()
    BroadcastData.status = .writing
    sessionBeginAtSourceTime = nil
                    
    configurateVideoWriter()
}

private func configurateVideoWriter() {
    let outputFileLocation = videoService.getVideoFileLocation()
    
    videoWriter = try? AVAssetWriter.init(outputURL: outputFileLocation,
                                          fileType: AVFileType.mov)
    
    configurateVideoWriterInput()
    configurateMicrophoneWriterInput()
    
    if videoWriter.canAdd(videoWriterInput) { videoWriter.add(videoWriterInput) }
    if videoWriter.canAdd(microphoneWriterInput) { videoWriter.add(microphoneWriterInput) }

    videoWriter.startWriting()
}

private func configurateVideoWriterInput() {
    let RESOLUTION_COEF: CGFloat = 16
    let naturalWidth = UIScreen.main.bounds.width
    let naturalHeight = UIScreen.main.bounds.height
    let width = naturalWidth - naturalWidth.truncatingRemainder(dividingBy: RESOLUTION_COEF)
    let height = naturalHeight - naturalHeight.truncatingRemainder(dividingBy: RESOLUTION_COEF)
    
    let videoSettings: [String: Any] = [
        AVVideoCodecKey: AVVideoCodecType.h264,
        AVVideoWidthKey: width,
        AVVideoHeightKey: height
    ]
    
    videoWriterInput = AVAssetWriterInput(mediaType: .video,
                                          outputSettings: videoSettings)
    videoWriterInput.expectsMediaDataInRealTime = true
}

private func configurateMicrophoneWriterInput() {
    let audioOutputSettings: [String : Any] = [
        AVFormatIDKey: kAudioFormatMPEG4AAC,
        AVNumberOfChannelsKey : 1,
        AVSampleRateKey : 44100.0,
        AVEncoderBitRateKey: 96000
    ]

    microphoneWriterInput = AVAssetWriterInput(mediaType: .audio,
                                               outputSettings: audioOutputSettings)
    microphoneWriterInput.expectsMediaDataInRealTime = true
}

3.Write process:

override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: 
RPSampleBufferType) {
    guard isRecording && videoWriter?.status == .writing else { return }
    
    if BroadcastData.status != .writing {
        isRecording = false
        finishBroadCast()
        return
    }
    
    if sessionBeginAtSourceTime == nil {
        sessionBeginAtSourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
        videoWriter.startSession(atSourceTime: sessionBeginAtSourceTime!)
    }
    
    switch sampleBufferType {
    case .video:
        if videoWriterInput.isReadyForMoreMediaData {
            videoWriterInput.append(sampleBuffer)
        }
    case .audioMic:
        if microphoneWriterInput.isReadyForMoreMediaData {
            microphoneWriterInput.append(sampleBuffer)
        }
    case .audioApp:
        break
    @unknown default:
        print("unknown")
    }
}

4.Pause and resume

override func broadcastPaused() {
    super.broadcastPaused()
}

override func broadcastResumed() {
    super.broadcastResumed()
}

Answer 1

Pausing and resuming the recording creates a gap in the video presentation timestamps and a discontinuity in the audio, which I believe explains your symptoms.

What you need to do is measure how long the recording was paused for, possibly using the sample buffer timestamps, and then subtract that offset from the presentation timestamps of all the subsequent CMSampleBuffer 's that you process. CMSampleBufferCreateCopyWithNewTiming() can help you with this.