人体姿势估计/匹配在智能手机上

Question

Im working on a project where a person must mimic a predefined pose. A picture is made from the person that mimics this predefined pose. Then, the human pose of the person is extracted from this image and compared with the predefined pose. Finally a scoring mechanism decides how well the two poses match or if they match at all.

I want to develop for smartphone, so ideally everything runs embedded on the smartphone itself. This means, the implementation is capable of running on CPU or smartphone GPU (example Moto G5 plus, Adreno 506 GPU on board -supports OpenGL-). Working embedded is not a must, i think it's also possible to outsource the estimation/matching algorithm to a central server containing a decent GPU. This particular choice, embedded or out-sourcing, is an issue that involves a lot of parameters (performance/computation power, server cost, accuracy, mobile battery usage, delay server communication, multi platform, scalability, mobile data usage -less important- , ...)

I know there are some frameworks out there for human pose estimation, like Openpose and deepercut. But as they all use deep learning, they require a descent GPU. Most of the new smartphones these days have a GPU on board, but are they capable of running these frameworks? To nuance for this case, the (multi-person) keypoint detection doesn't need to be realtime, as there is only 1 picture (no realtime video) and a delay time of 2 to 5 seconds is acceptable.

As I'm still in the research phase, I don't know what direction I should go. Is it even possible to port these frameworks to a smartphone platform? Like Openpose for example, which uses Caffe and OpenCV. Let's say I want to port Openpose to Android; I know there is a library CNNdroid that is capable of converting CNN models made with Caffe to CNNdroid format. Further OpenCV also shouldn't be a big problem as there is a Android version available. So, in theory it seems possible, but what in practice..

My question is: Is there someone who has experience with human pose detection/matching on smartphone? Is it even possible with the current GPU's available on smartphone. I know this is a broad question, but some directions/suggestions/experience could really help

UPDATE: I'm thinking about the option of porting Openpose (uses Caffe as ML framework) to TensorFlow. TensorFlow supports both Android & iOS

Answer 1

You might be interested in looking at the techniques used by Krafka et al. for their Eye Tracking for Everyone project in which they compress a larger network for estimating gaze coordinates into a smaller network which can run on a smartphone. This is using a concept developed by Geoff Hinton which he called Dark Knowledge . Gaze detection is a special case of pose estimation, so in principle it would seem like these techniques would be helpful. However, I do not know whether they will be sufficiently effective for your purposes (I think that largely depends on your accuracy constraints).

Answer 2

Openpose is too heavy for a smartphone application. You need to redesign the software architecture to fit into a phone system. Regarding to the cnn which is the bottleneck of the performance, using mobilenet like structure and Dark Knowledge (as Mozglubov mentioned) to teach a thinner network are two promising approaches. Anywhere, there are lots of engineer work ahead. Good luck!