Object Detection on Android using Tensorflow 2

Question

I am trying to apply a custom object detection model on android. To apply the model I use the tensorflow repository example under /lite/examples/object_detection. For this reason I am also using my personal phone (Xiaomi Redmi Note 8 pro, Android 10) for testing. The example works perfectly, being able to identify the different objects. However, when I try to import my custom model the applycation crashes repeatedly. To run the model I have added at build.gradle(:app) the following dependencies:

implementation('org.tensorflow:tensorflow-lite:0.0.0-nightly') { changing = true }
implementation('org.tensorflow:tensorflow-lite-gpu:0.0.0-nightly') { changing = true }
implementation('org.tensorflow:tensorflow-lite-support:0.0.0-nightly') { changing = true }

as well as I have replaced the detect.tflite and labelmap.txt with my files. Additionally, I attempled to change the DetectorActivity.java ( TF_OD_API_INPUT_SIZE and TF_OD_API_IS_QUANTIZED ) because the model I use( ssd_mobilenet_v2_fpnlite _640x640_coco17_tpu-8) has a different input size, but it still crashes. There is a guidance Running TF2 Detection API Models on mobile , but I haven't manage to achieve sth yet. Am I doing something wrong with the procedure on Android or the model is the problem? Thank you!

I am adding the Logcat output of the crash:

2020-10-06 19:30:28.163 25354-25354/? E/mples.detectio: Unknown bits set in runtime_flags: 0x8000
2020-10-06 19:30:28.407 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "ro.vendor.df.effect.conflict"
2020-10-06 19:30:28.560 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.560 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.560 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.560 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.560 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.560 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.560 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.560 25354-25372/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.561 25354-25372/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.561 25354-25372/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.561 25354-25372/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "camera.aux.packagelist"
2020-10-06 19:30:28.626 25354-25354/org.tensorflow.lite.examples.detection E/GraphicExt: Can't load libboost_ext_fwk
2020-10-06 19:30:28.627 25354-25354/org.tensorflow.lite.examples.detection E/GraphicExt: GraphicExtModuleLoader::CreateGraphicExtInstance false
2020-10-06 19:30:28.642 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "vendor.debug.bq.dump"
2020-10-06 19:30:28.642 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "vendor.debug.bq.dump"
2020-10-06 19:30:28.642 25354-25354/org.tensorflow.lite.examples.detection E/libc: Access denied finding property "vendor.debug.bq.dump"
2020-10-06 19:30:28.642 25354-25354/org.tensorflow.lite.examples.detection E/GraphicExt: GraphicExtModuleLoader::CreateGraphicExtInstance false
2020-10-06 19:30:28.657 25354-25354/org.tensorflow.lite.examples.detection E/AndroidRuntime: FATAL EXCEPTION: main
    Process: org.tensorflow.lite.examples.detection, PID: 25354
    java.lang.IllegalStateException: This model does not contain associated files, and is not a Zip file.
        at org.tensorflow.lite.support.metadata.MetadataExtractor.assertZipFile(MetadataExtractor.java:325)
        at org.tensorflow.lite.support.metadata.MetadataExtractor.getAssociatedFile(MetadataExtractor.java:165)
        at org.tensorflow.lite.examples.detection.tflite.TFLiteObjectDetectionAPIModel.create(TFLiteObjectDetectionAPIModel.java:116)
        at org.tensorflow.lite.examples.detection.DetectorActivity.onPreviewSizeChosen(DetectorActivity.java:99)
        at org.tensorflow.lite.examples.detection.CameraActivity$7.onPreviewSizeChosen(CameraActivity.java:446)
        at org.tensorflow.lite.examples.detection.CameraConnectionFragment.setUpCameraOutputs(CameraConnectionFragment.java:357)
        at org.tensorflow.lite.examples.detection.CameraConnectionFragment.openCamera(CameraConnectionFragment.java:362)
        at org.tensorflow.lite.examples.detection.CameraConnectionFragment.access$300(CameraConnectionFragment.java:66)
        at org.tensorflow.lite.examples.detection.CameraConnectionFragment$3.onSurfaceTextureAvailable(CameraConnectionFragment.java:171)
        at android.view.TextureView.getTextureLayer(TextureView.java:406)
        at android.view.TextureView.draw(TextureView.java:349)
        at android.view.View.updateDisplayListIfDirty(View.java:20568)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.updateDisplayListIfDirty(View.java:20559)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.updateDisplayListIfDirty(View.java:20559)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.draw(View.java:21740)
        at android.view.View.updateDisplayListIfDirty(View.java:20568)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at androidx.coordinatorlayout.widget.CoordinatorLayout.drawChild(CoordinatorLayout.java:1246)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.draw(View.java:21740)
        at android.view.View.updateDisplayListIfDirty(View.java:20568)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.updateDisplayListIfDirty(View.java:20559)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.updateDisplayListIfDirty(View.java:20559)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.updateDisplayListIfDirty(View.java:20559)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.updateDisplayListIfDirty(View.java:20559)
        at android.view.View.draw(View.java:21448)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4477)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4237)
        at android.view.View.draw(View.java:21740)
        at com.android.internal.policy.DecorView.draw(DecorView.java:844)
        at android.view.View.updateDisplayListIfDirty(View.java:20568)
        at android.view.ThreadedRenderer.updateViewTreeDisplayList(ThreadedRenderer.java:575)
        at android.view.ThreadedRenderer.updateRootDisplayList(ThreadedRenderer.java:581)
        at android.view.ThreadedRenderer.draw(ThreadedRenderer.java:654)
2020-10-06 19:30:28.658 25354-25354/org.tensorflow.lite.examples.detection E/AndroidRuntime:     at android.view.ViewRootImpl.draw(ViewRootImpl.java:3877)
        at android.view.ViewRootImpl.performDraw(ViewRootImpl.java:3668)
        at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:2988)
        at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:1888)
        at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:8043)
        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:969)
        at android.view.Choreographer.doCallbacks(Choreographer.java:793)
        at android.view.Choreographer.doFrame(Choreographer.java:728)
        at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:954)
        at android.os.Handler.handleCallback(Handler.java:914)
        at android.os.Handler.dispatchMessage(Handler.java:100)
        at android.os.Looper.loop(Looper.java:224)
        at android.app.ActivityThread.main(ActivityThread.java:7551)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:539)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:995)

Update: Using a different device (Xiaomi Redmi Note 4X, Android 7) the custom application works, without being able to identify customs objects with bouding boxes though. So, for sure there is a problem for Android 10 for me and for the second case something with the labelmap.txt file I guess (if not with the trained model).

Answer 1

To answer my question, the main issue regarding the implementation of the custom object detection model was the absence of a metadata file that should be attached to the model file(.tflite in my case). In other words, a file that describes and specifies which is the model, the resolution of the input images in the android application, the label file that you would like to use etc. (So the application was expecting a 300x300 resolution rather than a 640x640 which was the one I wanted to feed and that was the reason of crashing). For more information you can check a relevant question: here .