SceneKit：如何从 3D 模型中识别和访问人脸？

Question

In Blender, you can see and access each face of a 3D model like this one: https://poly.google.com/view/6mRHqTCZHxw

Is it possible in SceneKit to do the same, that is access each face of the model?

This question is similar and implies it is impossible, but does not confirm if SceneKit lets you programmatically access all faces of a model. (It focuses on identifying the face touched.)

Two questions:

1) Can you programmatically access each face?

2) Can you filter and only access faces that are visible (ie, ignore faces that "inside" or occluded by other faces)?

Answer 1

An implementation of @Xartec's answer for your first question #1, based also on the Apple documentation, in Swift 5.3:

extension SCNGeometryElement {
    var faces: [[Int]] {
        func arrayFromData<Integer: BinaryInteger>(_ type: Integer.Type, startIndex: Int = 0, size: Int) -> [Int] {
            assert(self.bytesPerIndex == MemoryLayout<Integer>.size)
            return [Integer](unsafeUninitializedCapacity: size) { arrayBuffer, capacity in
                self.data.copyBytes(to: arrayBuffer, from: startIndex..<startIndex + size * MemoryLayout<Integer>.size)
                capacity = size
            }
                .map { Int($0) }
        }

        func integersFromData(startIndex: Int = 0, size: Int = self.primitiveCount) -> [Int] {
            switch self.bytesPerIndex {
            case 1:
                return arrayFromData(UInt8.self, startIndex: startIndex, size: size)
            case 2:
                return arrayFromData(UInt16.self, startIndex: startIndex, size: size)
            case 4:
                return arrayFromData(UInt32.self, startIndex: startIndex, size: size)
            case 8:
                return arrayFromData(UInt64.self, startIndex: startIndex, size: size)
            default:
                return []
            }
        }

        func vertices(primitiveSize: Int) -> [[Int]] {
            integersFromData(size: self.primitiveCount * primitiveSize)
                .chunked(into: primitiveSize)
        }

        switch self.primitiveType {
        case .point:
            return vertices(primitiveSize: 1)
        case .line:
            return vertices(primitiveSize: 2)
        case .triangles:
            return vertices(primitiveSize: 3)
        case .triangleStrip:
            let vertices = integersFromData(size: self.primitiveCount + 2)
            return (0..<vertices.count - 2).map { index in
                Array(vertices[(index..<(index + 3))])
            }
        case .polygon:
            let polygonSizes = integersFromData()
            let allPolygonsVertices = integersFromData(startIndex: polygonSizes.count * self.bytesPerIndex, size: polygonSizes.reduce(into: 0, +=))
            var current = 0
            return polygonSizes.map { count in
                defer {
                    current += count
                }
                return Array(allPolygonsVertices[current..<current + count])
            }
        @unknown default:
            return []
        }
    }
}

The resulting arrays is an array of faces, each faces containing a list of vertex index.
An answering for how to extract the vertices from SCNGeometrySource can be found there https://stackoverflow.com/a/66748865/3605958 , and can be updated to get colors instead.

You will need this extension that implements the chunked(into:) method used above:

extension Collection {
    func chunked(into size: Index.Stride) -> [[Element]] where Index: Strideable {
        precondition(size > 0, "Chunk size should be atleast 1")
        return stride(from: self.startIndex, to: self.endIndex, by: size).map {
            Array(self[$0..<Swift.min($0.advanced(by: size), self.endIndex)])
        }
    }
}

For #2, I don't believe there's a way.

Answer 2

You can, but there is no convenient way built into SceneKit that lets you do it so you would have to built that yourself.

1. Yes, if you define what a face is and map that to the vertices in the model. For example, you could read the SCNGeometry's SCNGeometrySources into your own arrays of face objects, in the same order. Using the faceIndex you can than get the index to your array of faces. To update them, you would have to construct a SCNGeometry based on SCNGeometrySources programmatically, based on your own data from the faces array.

Note, the faceIndex returns the triangle rendered and not the quad/polygon so you have to convert it (very doable if all quads).

I'm working on a SceneKit based app that is basically a mini Blender for ipad pros. It uses a halfedge datastructure with objects for vertices and edges and faces. This allows access to those elements but in reality it allows access to the half edge data structure mapped to the model, which forms the basis for the geometry that replaces the one rendered.

2. Not directly. If you have the geometry mapped to a data model it is of course possible to calculate it before rendering but unfortunately Scenekit doesn't provide a convenient way to know which faces weren't rendered.

That all said, a face is merely a collection of vertices and indices, which are stored in the SCNGeometrySources. It may be easier to provide a better answer if you add why you want to add the faces and what you want to do with its vertices.

EDIT: based on your comment "if they tap on face, for instance, the face should turn blue."

As I mentioned above, a face is merely a collection of vertices and indices, a face itself does not have a color, it is the vertices that can have a color. A SCNNode has a SCNGeometry that has several SCNGeometrySources that hold the information about the vertices and how they are used to render faces. So what you want to do is go from faceIndex to corresponding vertex indices in the SCNGeometrySource. You then need to read the latter into an array of vectors, update them as desired, and then create a SCNGeometrySource based on your own array of vectors.

As I mentioned the faceIndex merely provides an index of what was rendered an not necessarily what you fed it (the SCNGeometrySource) so this requires mapping the model to a data structure.

If your model would consists of all triangles and has unique verts opposed to shared, does not interleave the vertex data, then faceIndex 0 would correspond to vertex 0, 1, and 2, and faceIndex 1 would correspond to vertex 3, 4, and 5 in the SCNGeometrySource. In case of quads and other polygons and interleaved vertex data it becomes significantly more complicated.

In short, there is no direct access to face entities in SceneKit but it is possible to modify the SCNGeometrySources (with vertex positions, colors, normals uv coords) programmatically.

EDIT 2: based on further comments: The primitiveType tells Scenekit how the model is constructed, it does not actually convert it. So it would still require the model to be triangulated already. But then if all triangles, AND if the model uses unique vertices (opposed to sharing verts with adjacent faces, model io provides a function to split vertices to unique from shared if necessary) AND if all the verts in the SCNGeometrySource are actually rendered (which is usually the case if the model is properly constructed), then yes. It is possible to do the same with polygons, see https://developer.apple.com/documentation/scenekit/scngeometryprimitivetype/scngeometryprimitivetypepolygon

Polygon 5, 3, 4, 3 would correspond to face index 0, 1, 2, 3 only if they were all triangles which they are obviously not. Based on the number of vertices per polygon however you can determine how many triangles will be rendered for the polygon. Based on that it is possible to get the index of the corresponding verts.

For example, the first polygon corresponds to face index 0, 1 and 2 (takes 3 triangles to create that polygon with 5 verts), the second polygon is face index 3, the third polygon is faceIndex 4 and 5.

In practice that means looping through the polygons in the element and adding to a faceCounter var (increment with 1 for each vert more than 2) till you reached the same value as faceIndex. Though on my own datastructure, I actually do this same basic conversion myself and works quite well.

EDIT3: in practical steps:

Convert the SCNGeometryElement to an array of ints.

Convert the SCNGeometrySource with the color semantic to an array of vectors. It is possible there is no SCNGeometrySource with the color semantic in which case you will have to create it.

If the polygon primitive is used, loop through the first portion (up to the number of primitives, in this case polygons) of the array you created from the SCNGeometryElement and keep a counter to which you add 1 for every vert more than 2. So if the polygon has 3 verts, increment the counter with 1, if the polygon has 4 verts, increment with 2. Everytime you increment the counter, thus for every polygon, check if faceIndex has been reached. Once you get to the polygon that contains the tapped face, you can get the corresponding vertex indices from the second part of the SCNGeometryElement using the mapping depicted in the image above. If you add a second variable and increment that with the vertex count of each polygon while looping through them you already know the indices of the vertex indices stored in the element.

If all the polygons are quads the conversion is easier and faceindex 0 and 1 correspond to polygon 0, face index 2 and 3 to polygon 1.

Once you got the vertex indices from the SCNGeometryElement, you can modify the vertices at those indices in the array you created from and for the SCNGeometrySource. Then recreate and update the SCNGeometrySource of the SCNGeometry.

Last but not least, unless you use a custom shader, the vertex colors you provide through the SCNGeometrySource will only show up correctly if the material assigned has a white color as diffuse (so you may have to make the base texture white too).