How to determine (at runtime) if TensorFlow Lite is using a GPU or not?

Question

Is there a way for me to ensure or at least determine at runtime the correct accelerator (CPU, GPU) is used when using the TensorFlow Lite library?

Although I had followed the guide , and set the Interpreter.Options() object to use a GPU delegate on a device with a GPU (Samsung S9), its highly likely to be using the CPU in some cases. For example, if you use a quantized model with a default delegate options object, it will default to using the CPU, because quantizedModelsAllowed is set to false. I am almost sure that even though the options object passed to the interpreter had a GPUDelegate , that the CPU was used instead. Unfortunately, I had to guess based on speed of inference and accuracy.

There was no warning, I just noticed slower inference time and improved accuracy (because in my case the GPU was acting weirdly, giving me wrong values, and I am trying to figure out why as a separate concern). Currently, I have to guess if the GPU/ CPU is being used, and react accordingly. Now, I think there are other cases like this where it falls back to the CPU, but I don't want to guess.

---

I have heard of AGI (Android GPU Inspector), it currently only supports 3 pixel devices. It would have been nice to use this to see the GPU get used in the profiler. I have also tried Samsungs GPUWatch, this simply does not work (on both OpenGL and Vulkan), as my app doesn't use either of these APIs (it doesn't render stuff, it uses tensorflow!).

Answer 1

I will place my results here after using the benchmark tool:

Firstly you can see the model with CPU usage without XNNPack:

Secondly model with CPU with XNNPack:

Thirdly model with GPU usage!!!!!:

And lastly with Hexagon or NNAPI delegate:

As you can see model is been processed by GPU. Also I used 2 randomly selected phones. If you want any particular device please say it to me. Finally you can download all results from benchmark tool here .

Answer 2

Answer by TensorFlow advocate:

Q= What is happening when we set to use a specific delegate but the phone cannot support it? Let's say I set to use a Hexagon delegate and the phone cannot use it. It is going to fall back to CPU usage? What is happening when we set to use a specific delegate but the phone cannot support it? Let's say I set to use a Hexagon delegate and the phone cannot use it. It is going to fall back to CPU usage?

A= It should fallback to the CPU.

Q= What about if I set GPU and this delegate cannot support the specific model. Does it fall back to the CPU or it crashes? What about if I set GPU and this delegate cannot support the specific model. Does it fall back to the CPU or it crashes?

A= It should also fallback to CPU but the tricky thing is sometimes a delegate "thinks" it could support an op at initialization time, but during runtime "realize" that it can't support the particular configuration of the op in the particular model. In such cases, the delegate crashes.

Q= Is there a way to determine what delegate is used during runtime despite what we have set to use? A= You can look at the logcat, or use the benchmark tool to run the model on the particular phone to find out.

Answer 3

As Farmaker mentioned, TFLite's benchmarking & accuracy tooling is the best way for you to judge how a delegate will behave for your use-case (your model & device).

First, use the benchmark tool to check latencies with various configurations (for delegates, use params like use_gpu=true ). See this page for a detailed explanation of the tool, and pre-built binaries for you to use via adb. You can also use the param --enable_op_profiling=true to see which ops from the graph get accelerated by the delegate.

Then, if you want to check accuracy/correctness of a delegate for your model (ie whether the delegate behaves like CPU would numerically), look at this documentation for tooling details.