Face Features Detection - corner of eyes, eyebrows

Question

I am creating basic emotion detection system for mobile phone with usage of OpenCV4Android. My system is already capable of finding mouth and doing some preprocessing. I have nice results of getting face objects from Canny:

Examplary Face1: https://dl.dropboxusercontent.com/u/108321090/FACE%20%282%29.png

Examplary Face2: https://dl.dropboxusercontent.com/u/108321090/FACE%20%281%29.png

Red rectangles are areas found by cascades. I have those saved as Mat objects.

Blue dots are points I need to find. Problem is, that I have both eyebrows and eyes on the same segment.

Additionaly there are situations in which eyebrows are directly connected to eyes (in some emotion states). It's hard to access some points. I have also normal images (of course) and tresholded ones which are also interesting for eyebrow shapes - but I lose some other objects (mouth - well that one doesn't matter cuz its already done, eyes) due to bad light, well eyebrows are always well visible. Of course I could change tresholding a bit, cuz I dont need it in finding other features. Like I said mouths is done well. Eyes/Eyebrows left.

Examplary Face3: https://dl.dropboxusercontent.com/u/108321090/Screenshot_2014-01-17-01-33-14.png

Examplary Face4: https://dl.dropboxusercontent.com/u/108321090/Screenshot_2014-01-17-01-26-33.png

Examplary Face5 (a bit problematic, eyes gone, but if I treshold them localy not globaly its fine) https://dl.dropboxusercontent.com/u/108321090/Screenshot_2014-03-05-01-30-48.png

Exampalary Face6 (eyebrows conencted to eyes) https://dl.dropboxusercontent.com/u/108321090/Screenshot_2014-03-05-01-28-21.png

I want to ask you if you could provide me with any materials/ideas connected to detection of eye, and eyebrows action units.

Answer 1

if you can locate an eye/eye-brow unit you can probably just track it and relate emotions to the relative motion there rather than trying to separate eyes from eye-brows. Your first two exemplary faces are gradients while the rest are thresholded grey tones. I would rather use gradients since grey tones are affected by lighting and shadows.

I would also avoid using Canny edge detector since it is a highly non-linear and non-stable operator for matching sequential frames and hence for motion detection. I would rather use a simpler Sobel and some kind of motion detection but only after tracking subtracts a global head motion.

The interesting work on emotion detection was done based on Kinect and it really works though it requires a bit of offline training, see faceShift . A good test for right processing (before mapping features to emotions) is trying to move the model of the face in sync with target face - some kind of virtual avatar.