Automate Face detection testing with Firebase ML Kit

Question

Currently we are working on a project using Firebase ML Kit to detect faces in frames sent from the camera. We would like to test how well the pre-trained model works on our own test data, and we tried to write a test for that that would run the model on all the test images, and then we can calculate some kind of accuracy .

Right now we are running an Android application and tried to make tests using Robolectric , and run the tests using Android sdk 28 . We read in the images one by one and convert them to byte arrays, create FirebaseVisionImages from them and then call detectInFace(image) .

The problem is that the detectInFace(image) uses callback functions to return the results. We checked if the callback function is ever called, but it doesn't seem like it. We tried to wait for the task to finish by using await(task) but it turned out that the task seems to never finish.

Does anyone know why the callback functions are never called/ seem to be never called? Are there any other ways to get the accuracy on custom images using test suites?

The code we have:

 FirebaseApp.initializeApp(InstrumentationRegistry.getInstrumentation().targetContext)
 val faceDetectorOptions: FirebaseVisionFaceDetectorOptions = FirebaseVisionFaceDetectorOptions.Builder()
            .setPerformanceMode(FirebaseVisionFaceDetectorOptions.FAST)
            .setContourMode(FirebaseVisionFaceDetectorOptions.NO_CONTOURS)
            .enableTracking()
            .build()

        // facedetector object with specified settings
        val faceDetector =
            FirebaseVision.getInstance()
                .getVisionFaceDetector(faceDetectorOptions)

        val metadata = FirebaseVisionImageMetadata.Builder()
            .setWidth(720) // 480x360 is typically sufficient for
            .setHeight(720) // image recognition
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
            .setRotation(FirebaseVisionImageMetadata.ROTATION_0)
            .build()

// Convert image to byteArray
        val inputStream =
          InstrumentationRegistry.getInstrumentation().targetContext.getAssets()?.open("testImage1.jpg")
        val bufferedStream = BufferedInputStream(inputStream);
        val bmp = BitmapFactory.decodeStream(bufferedStream)
        val outstream = ByteArrayOutputStream()
        bmp.compress(Bitmap.CompressFormat.PNG, 100, outstream)
        val bytearr = outstream.toByteArray()

      // detect the face in the image
        val task =  faceDetector.detectInImage(FirebaseVisionImage.fromByteArray(bytearr, metadata))
                .addOnSuccessListener { faces ->
                   // This seems to never be called
                }
                .addOnFailureListener {
                    // this also is never called
                }
        }
    } 

EDIT: Thanks for the answer and suggestion, In the end I fixed the problem by using FirebaseVisionImage.fromBitmap() instead of FirebaseVisionImage.fromByteArray() . No idea why but whenever I tried to run the model from a ByteArray it was getting 'stuck', but when running it from a bitmap it worked fine!

Answer 1

The problem is that all ml inferences happen in a worker thread and Robolectric shares a single thread for both UI operations and Test code.

You can call "shadowOf(getMainLooper()).idle();" to execute tasks getting posted to the worker thread in your test case.

See http://robolectric.org/blog/2019/06/04/paused-looper/