Kinect v2 Hand to Mouse position drops on hand close

Question

I am developing a WPF App that uses Kinect v2, and I use the hand to simulate the mouse. It works but I have a little problem - when I close the hand I simulate a click but the cursor drops its position a little bit relative to when the hand was open and sometimes it will end in a click in the wrong button or place.

Any ideas on how can I solve this?

I already tried to track the wrist and the thumbs instead of the hand but the problem still happens.

Thanks!

Answer 1

Here are some ideas:

• Filter and smooth the hand position data a bit more. For a UI/menu system, it should be acceptable to have some latency as it doesn't require reduced latency as much as other uses.
• Modify the hand position based on the hand's open/close state. Introduce a constant to bump up the hand position when the hand is closed, with appropriate smoothing to get this to feel and look correct
• Keep a list of hand positions and use the data from a few frames before (though it might be tricky to get this to feel and look correct)

As a note, also consider these points:

• Use bigger buttons. Buttons should have appropriate spacing, placement, and sizes. The app's UI should be specifically designed for a Kinect application.
• Use a different gesture for a mouse click, such as push or press which is the recommended approach in the Kinect Human Interface Guidelines 2.0