FPS not consistent on Camera using AVAssetWriter and CoreML

Question

I'm trying to create an app that can record video at 100 FPS using AVAssetWriter AND detect if a person is performing an action using the ActionClassifier from Create ML. But when I try to put the 2 together the FPS drops to 30 when recording and detecting actions.

If I do the recording by itself then it records at 100 FPS.

I am able to set the FPS of the camera to 100 FPS through the device configuration.

Capture output Function is setup

  func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        
        bufferImage = sampleBuffer
        guard let calibrationData = CMGetAttachment(sampleBuffer, key: kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, attachmentModeOut: nil) as? Data else {
            return
        }
        
        cameraCalibrationMatrix = calibrationData.withUnsafeBytes { $0.pointee }
        
        if self.isPredictorActivated == true {
            do  {
                let poses = try predictor.processFrame(sampleBuffer)
                if (predictor.isReadyToMakePrediction) {
                   let prediction =  try predictor.makePrediction()
                    let confidence = prediction.confidence * 100

                    DispatchQueue.main.async {
                        self.predictionLabel.text = prediction.label + " " + String(confidence.rounded(toPlaces: 0))
                        if (prediction.label == "HandsUp" && prediction.confidence > 0.85) {
                            print("Challenging")
                            self.didChallengeVideo()
                        }
                    }

                }
            } catch {
                print(error.localizedDescription)
            }
        }
        
       
        
        
        let presentationTimeStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
        
        if assetWriter == nil {
            createWriterInput(for: presentationTimeStamp)
        } else {
            let chunkDuration = CMTimeGetSeconds(CMTimeSubtract(presentationTimeStamp, chunkStartTime))
            //            print("Challenge\(isChallenging)")
            
            if chunkDuration > 1500 || isChallenging {
                assetWriter.endSession(atSourceTime: presentationTimeStamp)
                
                // make a copy, as finishWriting is asynchronous
                let newChunkURL = chunkOutputURL!
                let chunkAssetWriter = assetWriter!
                
                chunkAssetWriter.finishWriting {
                    print("finishWriting says: \(chunkAssetWriter.status.rawValue) \(String(describing: chunkAssetWriter.error))")
                    print("queuing \(newChunkURL)")
                    print("Chunk Duration: \(chunkDuration)")
                    
                    let asset = AVAsset(url: newChunkURL)
                    
                    print("FPS of CHUNK \(asset.tracks.first?.nominalFrameRate)")
                    
                    if self.isChallenging {
                        self.challengeVideoProcess(video: asset)
                    }
                    
                    self.isChallenging = false
                    
                    
                }
                createWriterInput(for: presentationTimeStamp)
                
            }
        }
        
        if !assetWriterInput.append(sampleBuffer) {
            print("append says NO: \(assetWriter.status.rawValue) \(String(describing: assetWriter.error))")
        }
        
        

    }

Answer 1

Performing action classification is quite expensive if you want to run it every frame so it may affect overall performance of the app (including video footage FPS). I don't know how often you need prediction but I would suggest you to try running Action Classifier 2-3 times per second maximum and see if that helps.

Running action classifier every frame won't change your classification that much because you're adding just one frame to your classifier action window so there is no need to run it so often.

For example if your action classifier was setup with window 3s and trained on 30fps videos, your classification is based on 3 * 30 = 90 frames. One frame won't make a difference.

Also make sure that your 100fps matches footage that you used for training action classifier. Otherwise you can get wrong predictions because running Action Classifier trained on 30fps video will treat 1s of 100fps footage as more than 3,333s.