Capturing video at 60 FPS with the front camera on iPad Pro 9.7"

Question

I am trying to configure the AVCaptureSession to grab frames at 640x480 at 60 FPS with the front camera on iPad Pro 9.7".

During the configuration I have these lines:

  for videoFormat in (captureDevice?.formats)! {

    let descriptions = videoFormat.formatDescription
    let dimensions = CMVideoFormatDescriptionGetDimensions(descriptions)
    let width = dimensions.width
    let height = dimensions.height

    if ((width != 640) || (height != 480)) {
      continue
    }
    //      let videoFormatDescriptionRef = videoFormat.formatDescription

    for range in videoFormat.videoSupportedFrameRateRanges {
      if range.maxFrameRate == 60 {
        bestFormat = videoFormat
        bestFrameRateRange = range
        break
      }
    }

    if ( bestFormat != nil) {
      captureDevice!.activeFormat = bestFormat!;
      captureDevice!.activeVideoMinFrameDuration = bestFrameRateRange!.minFrameDuration;
      captureDevice!.activeVideoMaxFrameDuration = bestFrameRateRange!.minFrameDuration;
      break
    }
  }

When I list the video formats supported by the camera I get this list:

<AVCaptureDeviceFormat: 0x282fa1fc0 'vide'/'420v'  192x 144, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @6.67), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

<AVCaptureDeviceFormat: 0x282fa1fb0 'vide'/'420f'  192x 144, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @6.67), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR, supports wide color> 

<AVCaptureDeviceFormat: 0x282fa1fa0 'vide'/'420v'  352x 288, { 2- 60 fps}, fov:49.895, max zoom:60.00 (upscales @3.33), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

<AVCaptureDeviceFormat: 0x282fa1f90 'vide'/'420f'  352x 288, { 2- 60 fps}, fov:49.895, max zoom:60.00 (upscales @3.33), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR, supports wide color> 

<AVCaptureDeviceFormat: 0x282fa1f80 'vide'/'420v'  480x 360, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @2.67), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

<AVCaptureDeviceFormat: 0x282fa1f70 'vide'/'420f'  480x 360, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @2.67), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR, supports wide color> 

<AVCaptureDeviceFormat: 0x282fa1f60 'vide'/'420v'  640x 480, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @2.00), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

<AVCaptureDeviceFormat: 0x282fa1f50 'vide'/'420f'  640x 480, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @2.00), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR, supports wide color> 

<AVCaptureDeviceFormat: 0x282fa2030 'vide'/'420v'  960x 540, { 2- 60 fps}, fov:58.480, max zoom:48.50 (upscales @1.43), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

<AVCaptureDeviceFormat: 0x282fa2040 'vide'/'420f'  960x 540, { 2- 60 fps}, fov:58.480, max zoom:48.50 (upscales @1.43), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR, supports wide color> 

<AVCaptureDeviceFormat: 0x282fa2050 'vide'/'420v' 1024x 768, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @1.25), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

<AVCaptureDeviceFormat: 0x282fa2060 'vide'/'420f' 1024x 768, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @1.25), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR, supports wide color> 

<AVCaptureDeviceFormat: 0x282fa2070 'vide'/'420v' 1280x 720, { 2- 60 fps}, fov:58.480, max zoom:48.50 (upscales @1.08), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

<AVCaptureDeviceFormat: 0x282fa2080 'vide'/'420f' 1280x 720, { 2- 60 fps}, fov:58.480, max zoom:48.50 (upscales @1.08), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR, supports wide color> 

<AVCaptureDeviceFormat: 0x282fa2090 'vide'/'420v' 1280x 960, { 2- 60 fps}, HRSI:2576x1932, fov:54.400, max zoom:60.00 (upscales @1.00), ISO:34.0-2176.0, SS:0.000013-0.500000> 

<AVCaptureDeviceFormat: 0x282fa20a0 'vide'/'420f' 1280x 960, { 2- 60 fps}, HRSI:2576x1932, fov:54.400, max zoom:60.00 (upscales @1.00), ISO:34.0-2176.0, SS:0.000013-0.500000, supports wide color> 

Because I am particularly interested in 640x480, I will want one of these:

<AVCaptureDeviceFormat: 0x282fa1f60 'vide'/'420v'  640x 480, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @2.00), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

<AVCaptureDeviceFormat: 0x282fa1f50 'vide'/'420f'  640x 480, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @2.00), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR, supports wide color> 

I choose the first one:

<AVCaptureDeviceFormat: 0x282fa1f60 'vide'/'420v'  640x 480, { 2- 60 fps}, fov:54.400, max zoom:60.00 (upscales @2.00), ISO:34.0-2176.0, SS:0.000013-0.500000, supports HDR> 

If I understood this correctly, if I choose this format I can set any frame rate I want between 2 and 60 FPS.

I choose 60 FPS, by setting these lines:

  captureDevice!.activeVideoMinFrameDuration = bestFrameRateRange!.minFrameDuration;
  captureDevice!.activeVideoMaxFrameDuration = bestFrameRateRange!.minFrameDuration;

I have even tried this

let fps60 = CMTimeMake(value: 1, timescale: 60)
captureDevice!.activeVideoMinFrameDuration = fps60;
captureDevice!.activeVideoMaxFrameDuration = fps60;

Whatever I do, video is always captured at 30 FPS.

I am setting the capture session with

self.captureSession.sessionPreset = AVCaptureSession.Preset.vga640x480
let captureDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: AVMediaType.video, position: .front)

Note: I am not particularly interested by 640x480. It only seemed to be the lightest resolution I could think that would make capture faster. I would go for any resolution that gives me 60 FPS on the front camera.

How can I do that?

Thanks

Answer 1

The following seems to be incorrect, because bestFrameRateRange!.minFrameDuration would have been 2?

captureDevice!.activeVideoMinFrameDuration = bestFrameRateRange!.minFrameDuration;
captureDevice!.activeVideoMaxFrameDuration = bestFrameRateRange!.minFrameDuration;

Also, are you calling lockForConfiguration before you set the frame duration?

do {
    try captureDevice!.lockForConfiguration()
    let fps60 = CMTimeMake(value: 1, timescale: 60)
    captureDevice!.activeVideoMinFrameDuration = fps60;
    captureDevice!.activeVideoMaxFrameDuration = fps60;
    captureDevice!.unlockForConfiguration()
} catch {
    // handle error
}

Answer 2

After a couple of days I found the problem.

You cannot use this line

self.captureSession.sessionPreset = AVCaptureSession.Preset.vga640x480

after configuring the session or it will undo everything you do to some internal standard.

This line has to be one of the first things you do.