Android ML Kit Not Able to Label Image

Question

So I have an Android app where a user can take a picture of a bird using the camera app and it will classify the bird. I followed the documentation for label images with a custom model for Android and it's not working. I have this piece of code in my onActivityResult :

val source = ImageDecoder.createSource(this.contentResolver, Uri.fromFile(photoFile))
val birdBitmap= ImageDecoder.decodeBitmap(source)

val image = InputImage.fromBitmap(birdBitmap, 0)
val localModel = LocalModel.Builder().setAssetFilePath("model.tflite").build()
val customImageLabelerOptions = CustomImageLabelerOptions.Builder(localModel)
    .setConfidenceThreshold(0.5f)
    .setMaxResultCount(5)
    .build()
val imageLabeler =
    ImageLabeling.getClient(customImageLabelerOptions)
imageLabeler.process(image)
    .addOnSuccessListener { labels ->
        var highConf = -1.0f
        var highText = ""
        for (label in labels) {
            val text = label.text
            val confidence = label.confidence
            val index = label.index

            if(confidence > highConf) {
                highConf = confidence
                highText = text
            }
        }
        Log.d("PREDICTION", "$highText, $highConf")
    }
    .addOnFailureListener { e ->
        Log.d("FAIL", "$e")
    }

When the user takes a picture, the addOnFailureListener is called and this is the error I get:

com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks
at com.google.mlkit.common.sdkinternal.ModelResource.zza(com.google.mlkit:common@@16.0.0:28)
at com.google.mlkit.common.sdkinternal.zzn.call(Unknown Source:6)
at com.google.mlkit.common.sdkinternal.zzm.run(com.google.mlkit:common@@16.0.0:5)
at com.google.mlkit.common.sdkinternal.zzq.run(com.google.mlkit:common@@16.0.0:3)
at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zzd(com.google.mlkit:common@@16.0.0:24)
at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zza(com.google.mlkit:common@@16.0.0:30)
at com.google.mlkit.common.sdkinternal.zzj.run(Unknown Source:2)
At java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:764)
Caused by: java.lang.IllegalArgumentException: Unsupported bitmap config HARDWARE
at com.google.android.libraries.vision.visionkit.pipeline.zzbb.zza(com.google.mlkit:vision-internal-vkp@@17.0.0:56)
at com.google.mlkit.vision.vkp.PipelineManager.process(com.google.mlkit:vision-internal-vkp@@17.0.0:150)
at com.google.mlkit.vision.label.custom.internal.zzd.zza(com.google.mlkit:image-labeling-custom@@16.1.0:19)
at com.google.mlkit.vision.label.custom.internal.zzd.run(com.google.mlkit:image-labeling-custom@@16.1.0:112)
at com.google.mlkit.vision.common.internal.MobileVisionBase.zza(com.google.mlkit:vision-common@@16.0.0:23)
at com.google.mlkit.vision.common.internal.zzb.call(Unknown Source:4)
at com.google.mlkit.common.sdkinternal.ModelResource.zza(com.google.mlkit:common@@16.0.0:26)

Can anyone tell me how do I fix this?

Answer 1

Looking at the stacktrace, it seems like the input Bitmap is taken directly from the camera and it resides in-memory ( Bitmap.Config.HARDWARE ). ML Kit only supports bitmap of ARGB_8888 format so please try:

val newBitmap = myBitmap.copy(Bitmap.Config.ARGB_8888, myBitmap.isMutable())

When we develop this, we were thinking about file saved on device and loading it as ARGB_8888 instead of from the device camera. I will file an enhancement request to see if we can cover this use-case. Thanks for flagging!