How to increase Frames Per Second during Face Detection using ML Kit

Question

Recently I am working with Google ML Kit Face Detection library. I implemented it successfully and able to detect my face through that. Played with contour points to draw a small bounding box over forehead.

During face detection it provide the image frames from Image Proxy. No my issue here is I'm getting only 6-7 frames per second. I want to increase the frame count approx. 25-30 frames per second.

I tried to decrease the target resolution from 640x480 to half of it, then I face 2 problems:

1. Input Image quality is very bad.
2. Due to which output image looses its YUV value and provide greyish green image.

Please help me with this to increase frame count per second.

Below are the library and code which I am using to achieve this task.

implementation 'com.google.mlkit:face-detection:16.1.5'
implementation 'androidx.camera:camera-core:1.0.2'
implementation 'androidx.camera:camera-camera2:1.0.2'
implementation 'androidx.camera:camera-lifecycle:1.0.2'
implementation 'androidx.camera:camera-view:1.0.0-alpha32'

private fun getImageBitmap(mediaImage: Image): Bitmap {
    val yBuffer = mediaImage.planes[0].buffer // Y
    val vuBuffer = mediaImage.planes[2].buffer // VU

    val ySize = yBuffer.remaining()
    val vuSize = vuBuffer.remaining()

    val nv21 = ByteArray(ySize + vuSize)

    yBuffer.get(nv21, 0, ySize)
    vuBuffer.get(nv21, ySize, vuSize)

    val yuvImage = YuvImage(nv21, ImageFormat.NV21, mediaImage.width, mediaImage.height, null)
    val outputStream = ByteArrayOutputStream()
    //yuvImage.compressToJpeg(cropRect, 100, outputStream)
    yuvImage.compressToJpeg(Rect(0, 0, yuvImage.width, yuvImage.height), 100, outputStream)
    val imageBytes = outputStream.toByteArray()
    val bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
    val rotated = rotateBitmap(bitmap, 270F)
    val bos = ByteArrayOutputStream()
    rotated.compress(CompressFormat.JPEG, 100, bos) //100 is the best quality possibe
    val square = bos.toByteArray()
    return BitmapFactory.decodeByteArray(square, 0, square.size)
}

Please check the below image for reference

Answer 1

Have you tried to set the option to PERFORMANCE_MODE_FAST? it says it will increase the return FPS, actually I have the same issue as you but I havn't got a chance to try it out.

Answer 2

You are doing unnecessary format conversion in your code. Encoding the YUV image to JPEG then decoding back to RGB is going to take a toll on your performance.

CameraX provides a feature that integrates MLKit with CameraX. Please see MLKitAnalyzer . It also takes care of difficult tasks like coordinate transformation. You can take a look at the sample here: https://github.com/androidx/androidx/blob/androidx-main/camera/integration-tests/viewtestapp/src/main/java/androidx/camera/integration/view/MlKitFragment.kt

If you want to do it yourself, then I would suggest you to take a look at the source code of MLKitAnalyzer and do the same thing.

Answer 3

If you only want to detect faces within 2 meters from the camera(selfie usecase). You can try new Face mesh detection with BOUNDING_BOX_ONLY mode.

It is faster than old Face detection(fast mode):

"~14 ms on Pixel 3" vs "~60ms on Pixel 3"