Swift 4：如何使用ios11视觉框架从人脸界标点创建人脸图

Question

I am using the ios 11 vision framework to yield the face landmark points in real time. I am able to get the face landmark points and overlay the camera layer with the UIBezierPath of the face landmark points. However, I would like to get something like the bottom right picture. Currently I have something that looks like the left picture, and I tried looping through the points and adding midpoints, but I don't know how to generate all those triangles from the points. How would I go about generating the map on the right from the points on the left?

I'm not sure I can with all the points I have, not that it will help too much, but I also have points from the bounding box of the entire face. Lastly, is there any framework that would allow me to recognize all the points I need, such as openCV or something else, please let me know. Thanks!

Here is the code I've been using from https://github.com/DroidsOnRoids/VisionFaceDetection :

func detectLandmarks(on image: CIImage) {
    try? faceLandmarksDetectionRequest.perform([faceLandmarks], on: image)
    if let landmarksResults = faceLandmarks.results as? [VNFaceObservation] {

        for observation in landmarksResults {

            DispatchQueue.main.async {
                if let boundingBox = self.faceLandmarks.inputFaceObservations?.first?.boundingBox {
                    let faceBoundingBox = boundingBox.scaled(to: self.view.bounds.size)
                    //different types of landmarks



                    let faceContour = observation.landmarks?.faceContour
                    self.convertPointsForFace(faceContour, faceBoundingBox)

                    let leftEye = observation.landmarks?.leftEye
                    self.convertPointsForFace(leftEye, faceBoundingBox)

                    let rightEye = observation.landmarks?.rightEye
                    self.convertPointsForFace(rightEye, faceBoundingBox)

                    let leftPupil = observation.landmarks?.leftPupil
                    self.convertPointsForFace(leftPupil, faceBoundingBox)

                    let rightPupil = observation.landmarks?.rightPupil
                    self.convertPointsForFace(rightPupil, faceBoundingBox)

                    let nose = observation.landmarks?.nose
                    self.convertPointsForFace(nose, faceBoundingBox)

                    let lips = observation.landmarks?.innerLips
                    self.convertPointsForFace(lips, faceBoundingBox)

                    let leftEyebrow = observation.landmarks?.leftEyebrow
                    self.convertPointsForFace(leftEyebrow, faceBoundingBox)

                    let rightEyebrow = observation.landmarks?.rightEyebrow
                    self.convertPointsForFace(rightEyebrow, faceBoundingBox)

                    let noseCrest = observation.landmarks?.noseCrest
                    self.convertPointsForFace(noseCrest, faceBoundingBox)

                    let outerLips = observation.landmarks?.outerLips
                    self.convertPointsForFace(outerLips, faceBoundingBox)
                }
            }
        }
    }

}

func convertPointsForFace(_ landmark: VNFaceLandmarkRegion2D?, _ boundingBox: CGRect) {
    if let points = landmark?.points, let count = landmark?.pointCount {
        let convertedPoints = convert(points, with: count)



        let faceLandmarkPoints = convertedPoints.map { (point: (x: CGFloat, y: CGFloat)) -> (x: CGFloat, y: CGFloat) in
            let pointX = point.x * boundingBox.width + boundingBox.origin.x
            let pointY = point.y * boundingBox.height + boundingBox.origin.y

            return (x: pointX, y: pointY)
        }

        DispatchQueue.main.async {
            self.draw(points: faceLandmarkPoints)
        }
    }
}


func draw(points: [(x: CGFloat, y: CGFloat)]) {
    let newLayer = CAShapeLayer()
    newLayer.strokeColor = UIColor.blue.cgColor
    newLayer.lineWidth = 4.0

    let path = UIBezierPath()
    path.move(to: CGPoint(x: points[0].x, y: points[0].y))
    for i in 0..<points.count - 1 {
        let point = CGPoint(x: points[i].x, y: points[i].y)
        path.addLine(to: point)
        path.move(to: point)
    }
    path.addLine(to: CGPoint(x: points[0].x, y: points[0].y))
    newLayer.path = path.cgPath

    shapeLayer.addSublayer(newLayer)
}

Answer 1

I did end up finding a solution that works. I used delaunay triangulation via https://github.com/AlexLittlejohn/DelaunaySwift , and I modified it to work with the points generated via the vision framework's face landmark detection request. This is not easily explained with a code snippet, so I have linked my github repo below that shows my solution. Note that this doesn't get the points from the forehead, as the vision framework only gets points from the eyebrows down.

https://github.com/ahashim1/Face

Answer 2

What you want in the image on the right is a Candide mesh. You need to map these points to the mesh and that will be it. I don't think you need to go the route that has been discussed in the comments.

PS I found Candide while going through the APK contents of a famous filters app(reminds me of casper) - haven't had the time to try it myself yet.