Tracking hand rotation with RotatedRect

Question

I'm using OpenCV for a project to detect hand gestures, I managed to isolate the hand and drawing the convexes the hand is appearing very clearly now. My goal is to detect if the hand is rotating or moving left or right.

I used RotatedRect to figure out if the hand is rotated or not, but it seems like the RotatedRect is always tilted left or right (mostly left) even if my hand is straight. How can I possibly fix that?

Here's my code for creating and drawing the RotatedRect:

double angle;
            RotatedRect rc = minAreaRect(biggestcontours[hand]);
            if (rc.size.width < rc.size.height) {
                angle = 90 - rc.angle;
            }
            else {
                angle = -rc.angle;
            }
        Point2f rect_points[4];
            rc.points(rect_points);
            for (int j = 0; j < 4; j++)
            {
                line(onehand, rect_points[j], rect_points[(j + 1) % 4], Scalar(255, 0, 0));
            }

Remark :biggestcontours[hand] contains the contour of my hand.

Answer 1

Here is an approach using PCA

You can find the code here

Result:

Note: if the hand shape doesn't change and you have a reference angle, you'll be able to get it's rotation angle accurately.