How improve tflite_flutter performance

Question

I am using tflite_flutter in my app for

• a Dog Detector, which returns a Rect if there is a dog in the camera view
• a Dog Classifier, which crops the image using the Rect and returns the breed.

Both use tflite_flutter. Each takes 50 - 70 milliseconds on my Samsung Galaxy S10e. I want to improve the performance. I have tried

• varying the number of threads
• using..useNnApiForAndroid = true
• using..addDelegate(gpuDelegateV2) and..addDelegate(NnApiDelegate() after install.bat -d
• running the detector in an isolate

Nothing helps. What else can I try? Any ideas, anyone?

Answer 1

Inference/Latency of lite operations depends on multiple factors like delegate compatibility, whether delegates has been enabled by manufacturer or not etc.

Attached few factors to optimize the models for low latency and high inference time.

1. Quantization and the delegates compatibility: For eg. GPU delgates support all ranges of quantization but Hexagon delegates are more compatible with integer and quantization aware modesl

2. GPU compatibility of layers inside model. All ops are not supported for GPU delegates by default, so you have to put alternate apis to use GPU delegates. for eg leakyrelu is not supported but relu is supported for GPU.

you can check GPU compatibility of your lite model using model_analyzer , inference timing depends on how model is leveraging the GPU/NNAPI/Other delegates efficiently. You can use Benchamarking apk to check performance of lite models against different delegates, optimize the model accordingly by disabling quantization of suspected layers.

3. User Serialization and on-device Training: You can use serialization / on device training ** to reduce the warm-up time or improve inference time.

4. Same input shape during inference:

Please make sure you have used same input shape as lite model input during inference to avoid dimension mismatch issues.

Thank you!