在 CMSampleBuffer 中获取非矩形轮廓内 RGB 平均值的最快方法

Question

I am trying to get the RGB average inside of a non-rectangular multi-edge (closed) contour generated over a face landmark region in the frame (think of it as a face contour) from AVCaptureVideoDataOutput. I currently have the following code,

        let landmarkPath = CGMutablePath()
        let landmarkPathPoints = landmark.normalizedPoints
            .map({ landmarkPoint in
                CGPoint(
                    x: landmarkPoint.y * faceBoundingBox.height + faceBoundingBox.origin.x,
                    y: landmarkPoint.x * faceBoundingBox.width + faceBoundingBox.origin.y)
            })
        landmarkPath.addLines(between: landmarkPathPoints)
        landmarkPath.closeSubpath()

        let averageFilter = CIFilter(name: "CIAreaAverage", parameters: [kCIInputImageKey: frame, kCIInputExtentKey: landmarkPath])!
        let outputImage = averageFilter.outputImage!

However, it currently throws *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFType CGRectValue]: unrecognized selector sent to instance 0x283a57a80' terminating with uncaught exception of type NSException. I suspect this is as the kCIInputExtentKey is not a proper CIVector rectangular object. Is there anyway to fix this? How can I define a non-rectangular region for the CIAreaAverage filter? If not possible, what's the most efficient way of getting the average RGB across the region of interest?

Thanks a lot in advance!

Answer 1

If you could make all pixels outside of the contour transparent then you could use CIKmeans filter with inputCount equal 1 and the inputExtent set to the extent of the frame to get the average color of the area inside the contour (the output of the filter will contain 1-pixel image and the color of the pixel is what you are looking for).

Now, to make all pixels transparent outside of the contour, you could do something like this:

1. Create a mask image but setting all pixels inside the contour white and black outside (set background to black and fill the path with white).
2. Use CIBlendWithMask filter where:
   • inputBackgroundImage is a fully transparent (clear) image
   • inputImage is the original frame
   • inputMaskImage is the mask you created above

The output of that filter will give you the image with all pixels outside the contour fully transparent. And now you can use the CIKMeans filter with it as described at the beginning.

BTW, if you want to play with every single of the 230 filters out there check this app out: https://apps.apple.com/us/app/filter-magic/id1594986951