AVCaptureVideoDataOutputSampleBufferDelegate with SceneKit and AVDeviceCapture background

Question

I'm trying to grab the CVPixelBuffer while using the camera video feed as a background to SceneKit. Please note I cannot use ARKit for my project as I need the front + back camera and no world tracking overhead. Even attempting the most basic example:

let captureDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back)
self.scnScene.background.contents = captureDevice

results in the following error in iOS 11.2:

[SceneKit] Error: Cannot get pixel buffer (CVPixelBufferRef)

Whenever this error pops up, the AVCaptureVideoDataOutputSampleBufferDelegate stops being called.

Answer 1

Since I was unable to simultaneously use the AVCaptureDevice as a background to SceneKit and grab its CVPixelBuffer , my solution was to render the sample buffer to a MTLTexture then apply that as a background to SceneKit. See here as a starting point:

https://developer.apple.com/documentation/arkit/displaying_an_ar_experience_with_metal