Recognizing Sign Language in 3D Kinect

Question

We're making a sign language translator using kinect 1.0 device for my undergrad final year project. So far we have achieved recognizing gestures in 2D using the skeleton api's in kinect sdk and applied DTW algorithm on it. We also tracked fingers and distinguished between how many fingers are shown in the frame using contouring and applying convex hull on the contour. We used C# and Emgucv to achieve this.

Now we're stuck at how to transform the data into 3d coordniates. What I don't get is that:

1. How the 3d visualization will look like? I mean for now we just use the depth stream and apply a skin classifier on it to show only the skin parts as white pixels and the rest of the objects as black pixels, and we show the contoured and convex hulled area in the color stream. For 3d we'll use the same depth and color stream? If yes then how we'll transform the data and coordinates into 3d?

2. For gestures that involve touching of fingers on nose, how will I isolate the contoured area not to include all of the face and just to tell which finger touches which side of nose? Is this where 3d will come in?

3. Which api's and libraries are there that can help us in c#?

   Extracted Fingers after Contouring and Convex Hull

Answer 1

Kinect has support for creating a depth map using infrared lasers. It projects an infrared grid and measures the distance for each point in the grid. It seems that you're already using the depth info by this grid. For converting to 3D you should indeed use the depth info. Some basic trigonometry will help to transform the depth map into 3D (x,y,z) coordinates. The color stream from the camera can be mapped onto these points.

Detecting whether fingers are touching a nose is a difficult issue. While the grid density of the kinect is not very high, 3D probably won't help you. I would suggest to use edge detection (eg canny algorithm) with contour recognition on the camera images to detect whether a finger is in front of the face. Testing whether a finger actually touches the nose or is just close to is the real challenge.