Python + OpenCV face recognition

Question

I have one test1.jpg - it is the original image and contains the faces we will need to find.

My test2.jpg is some image to check. We will need check faces from test1.jpg in our test2.jpg and if some faces from test1.jpg is exist in test2.jpg output true .

I have not got prior experience in OpenCV and it new technology for me.

My very simple code:

# faces which we need find
known_image = cv2.imread('test1.jpg')
# some random image
unknown_image = cv2.imread('test2.jpg')

gray_known = cv2.cvtColor(known_image, cv2.COLOR_BGR2GRAY)
gray_unknown = cv2.cvtColor(unknown_image, cv2.COLOR_BGR2GRAY)
faceCascade = cv2.CascadeClassifier("haarcascade_frontalface_default.xml")

known_faces = faceCascade.detectMultiScale(
    gray_known,
    scaleFactor=1.27,
    minNeighbors=5,
    minSize=(10, 10),
    flags = cv2.CASCADE_SCALE_IMAGE
)

print("Found known {0} faces!".format(len(known_faces)))

unknown_faces = faceCascade.detectMultiScale(
    gray_unknown,
    scaleFactor=1.27,
    minNeighbors=5,
    minSize=(10, 10),
    flags = cv2.CASCADE_SCALE_IMAGE
)

print("Found unknown {0} faces!".format(len(unknown_faces)))

# BELLOW NOT CORRECT CODE. I DONT KNOW HOW I CAN DO LIKE THIS:
for face in known_faces:
   if face in unknown_faces:
      print(TRUE)
      break

Answer 1

The main issue with your code as it stands, is that it expects the face patches to be identical in two different images. This will almost certainly not be the case - even if one is simply a cropped image of another, if the cascade classifier finds the bounding box 1 pixel further to the left in one image than the other, the result is not identical.

The result of the cascade classifier is a set of bounding boxes that should represent faces (you'll sometimes get false detections). You then need to extract the sub-image within these bounding boxes and use some sort of comparison technique to work out if two face images are the same. I'll discuss these below.

Your problem actually has two stages, and it's important to differentiate them as there are different techniques to tackle the two problems.

Stage 1: Face Detection

This is the problem of actually finding faces in the images. Your code should handle this pretty well - the cascade classifier from OpenCV is one of the most common ways of doing this (an alternative is the aligned face detector in DLib ).

Stage 2: Face Recognition

This is the more difficult part. Face recognition techniques can vary greatly, and a lot of it is situation dependant. The most accurate of these uses the deep learning module , so if your hardware is up to the task I'd recommend that. It seems that code is actually an implementation of OpenFace - I've used it before and it's possible to get accurate recognitions on very few training images. I'd recommend reading the linked post, and the DNN samples as the code is probably too deep to go into here.

The main steps are as follows:

• Detect and locate faces in image 1 as above
• Use the pre-trained DNN to transform the face into a linearly separable feature vector (known as an embedding).
• Detect and locate faces in image 2 as above
• Again, transform the faces into the feature vector using the DNN.
• Compare the feature vectors - not the raw images, and select those for which the euclidean distance is low enough.

The DNN embedding step is the slow part here, but should still be quite efficient if you have a GPU available.

If, rather than doing direct image-image comparison, you were looking for a known set of faces, you could train an off-the-shelf linear classifier, such as an SVM on the actual people you want to look for, then use this on the feature vectors you find in the image to see if they fall into any of the classes.

Stage 2(a): Face Recognition Without Deep Learning

OpenCV also comes with a face recognition module, which makes use of more classical techniques that tend to be faster and more portable, but at the cost of accuracy.

There's a really good tutorial on it in the OpenCV documentation . Again, the code is too long to put directly into a post, but that link describes 3 methods for face detection. Fisherfaces and Eigenfaces both rely on looking at a number of examples of the same face and modelling them in a lower dimensional space where they can be compared for their underlying properties, not the raw pixel data.

I'm not sure either of those will work well for your use-case as you will only have one example of each face.

Using Local Binary Pattern (LBP) histograms will likely give you a better start-point. LBP's are a way of describing textural details of an image - there's more information in the tutorial I linked above. I'll again give some basic steps here to get you started:

• For each detected face, extract the small sub-image (region-of-interest) using the bounding box for that face.
• Break that image up into smaller "patches" - essentially break the image up into a uniform grid (for example, a 100x100 pixel ROI might be broken up into 10x10 patches of 10x10 pixels).
• For each patch, calculate the LBP histogram (essentially a list of numbers)
• Concatenate (join) those histograms together to get a representation of the face. This concatenated histogram becomes your feature vector to represent that particular face.
• You can then do the same for image 2.
• For all of your feature vectors from image 2, compare them against the ones from image 1 and try to pick out those with smaller distances.

Notes:

Any recognition technique using a single image is likely to be prone to error. I haven't tested the above algorithms, but have used OpenFace, Fisherfaces and LBPH for face recognition in the past, following those tutorials, and have found they work ok for recognition with a few examples of faces.