如何在 AR 场景中在运行时提取 SceneKit 深度缓冲区？

Question

How does one extract the SceneKit depth buffer? I make an AR based app that is running Metal and I'm really struggling to find any info on how to extract a 2D depth buffer so I can render out fancy 3D photos of my scenes. Any help greatly appreciated.

Answer 1

Your question is unclear but I'll try to answer.

Depth pass from VR view

If you need to render a Depth pass from SceneKit's 3D environment then you should use, for instance, a SCNGeometrySource.Semantic structure. There are vertex , normal , texcoord , color and tangent type properties. Let's see what a vertex type property is:

static let vertex: SCNGeometrySource.Semantic

This semantic identifies data containing the positions of each vertex in the geometry. For a custom shader program, you use this semantic to bind SceneKit's vertex position data to an input attribute of the shader. Vertex position data is typically an array of three- or four-component vectors.

Here's a code's excerpt from iOS Depth Sample project.

UPDATED: Using this code you can get a position for every point in SCNScene and assign a color for these points (this is what a zDepth channel really is):

import SceneKit

struct PointCloudVertex {
    var x: Float, y: Float, z: Float
    var r: Float, g: Float, b: Float
}

@objc class PointCloud: NSObject {

    var pointCloud : [SCNVector3] = []
    var colors: [UInt8] = []

    public func pointCloudNode() -> SCNNode {
        let points = self.pointCloud
        var vertices = Array(repeating: PointCloudVertex(x: 0,
                                                         y: 0,
                                                         z: 0,
                                                         r: 0,
                                                         g: 0,
                                                         b: 0), 
                                                     count: points.count)

        for i in 0...(points.count-1) {
            let p = points[i]
            vertices[i].x = Float(p.x)
            vertices[i].y = Float(p.y)
            vertices[i].z = Float(p.z)
            vertices[i].r = Float(colors[i * 4]) / 255.0
            vertices[i].g = Float(colors[i * 4 + 1]) / 255.0
            vertices[i].b = Float(colors[i * 4 + 2]) / 255.0
        }

        let node = buildNode(points: vertices)
        return node
    }

    private func buildNode(points: [PointCloudVertex]) -> SCNNode {
        let vertexData = NSData(
            bytes: points,
            length: MemoryLayout<PointCloudVertex>.size * points.count
        )
        let positionSource = SCNGeometrySource(
            data: vertexData as Data,
            semantic: SCNGeometrySource.Semantic.vertex,
            vectorCount: points.count,
            usesFloatComponents: true,
            componentsPerVector: 3,
            bytesPerComponent: MemoryLayout<Float>.size,
            dataOffset: 0,
            dataStride: MemoryLayout<PointCloudVertex>.size
        )
        let colorSource = SCNGeometrySource(
            data: vertexData as Data,
            semantic: SCNGeometrySource.Semantic.color,
            vectorCount: points.count,
            usesFloatComponents: true,
            componentsPerVector: 3,
            bytesPerComponent: MemoryLayout<Float>.size,
            dataOffset: MemoryLayout<Float>.size * 3,
            dataStride: MemoryLayout<PointCloudVertex>.size
        )
        let element = SCNGeometryElement(
            data: nil,
            primitiveType: .point,
            primitiveCount: points.count,
            bytesPerIndex: MemoryLayout<Int>.size
        )

        element.pointSize = 1
        element.minimumPointScreenSpaceRadius = 1
        element.maximumPointScreenSpaceRadius = 5

        let pointsGeometry = SCNGeometry(sources: [positionSource, colorSource], elements: [element])

        return SCNNode(geometry: pointsGeometry)
    }
}

Depth pass from AR view

If you need to render a Depth pass from ARSCNView it is possible only in case you're using ARFaceTrackingConfiguration for the front-facing camera. If so, then you can employ capturedDepthData instance property that brings you a depth map, captured along with the video frame.

var capturedDepthData: AVDepthData? { get }

But this depth map image is only 15 fps and of lower resolution than corresponding RGB image at 60 fps .

Face-based AR uses the front-facing, depth-sensing camera on compatible devices. When running such a configuration, frames vended by the session contain a depth map captured by the depth camera in addition to the color pixel buffer (see capturedImage ) captured by the color camera. This property's value is always nil when running other AR configurations.

And a real code could be like this:

extension ViewController: ARSCNViewDelegate {

    func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {

        DispatchQueue.global().async {

            guard let frame = self.sceneView.session.currentFrame else {
                return
            }
            if let depthImage = frame.capturedDepthData {
                self.depthImage = (depthImage as! CVImageBuffer)
            }
        }
    }
}

Depth pass from Video view

Also, you can extract a true Depth pass using 2 back-facing cameras and AVFoundation framework.

Look at Image Depth Map tutorial where Disparity concept will be introduced to you.