Three step threshold in Facenet model of face recogniton

Question

Suppose i trained the images of two people say Bob , Thomas .When i run the algorithm to detect the face of a totally different person from these two say John , then John is recognized as Bob or Thomas.How to avoid this ?

I am studying a face recognition model on GitHub(link) which uses Facenet model. Problem is when an unknown image (the image which is not in training data set) is given to identify , it identifies the unknown person as one the person in the data set .I searched on web and i found i need to increase threshold value .I guess i need to increase the threshold. But when i am increasing the threshold value to 0.99,0.99,99 then only it is rejecting the unknown image (image of the person who is not in data set) and sometimes even rejecting the image of person who is in dataset.

I guess by increasing the threshold value what we are assuring is that an image is classified as one of the person in training data only when they are close enough.

How to make changes so that the model works properly ?And can someone explain Threshold in Facenet model better.

Answer 1

The problem originated because of the nature of the code.

Code: https://github.com/AISangam/Facenet-Real-time-face-recognition-using-deep-learning-Tensorflow/blob/master/classifier.py

                model = SVC(kernel='linear', probability=True)
                model.fit(emb_array, label)

                class_names = [cls.name.replace('_', ' ') for cls in img_data]

As you see the code uses a SVC (Support Vector Classifier) to classify the classes. The SVC (or SVM) does not have an extra class for unknown class.

For the threshold variable, it is used in face detection, aka drawing a bounding box around the face for FaceNet to classify it.

Code:

https://github.com/AISangam/Facenet-Real-time-face-recognition-using-deep-learning-Tensorflow/blob/master/identify_face_image.py

            frame = frame[:, :, 0:3]
            bounding_boxes, _ = detect_face.detect_face(frame, minsize, pnet, rnet, onet, threshold, factor)
            nrof_faces = bounding_boxes.shape[0]

As you can see, the threshold variable is only used in detecting the bounding box.

Code for getting class name:

predictions = model.predict_proba(emb_array)
                    print(predictions)
                    best_class_indices = np.argmax(predictions, axis=1)
                    # print(best_class_indices)
                    best_class_probabilities = predictions[np.arange(len(best_class_indices)), best_class_indices]
                    print(best_class_probabilities)
                    cv2.rectangle(frame, (bb[i][0], bb[i][1]), (bb[i][2], bb[i][3]), (0, 255, 0), 2)    #boxing face

                    #plot result idx under box
                    text_x = bb[i][0]
                    text_y = bb[i][3] + 20
                    print('Result Indices: ', best_class_indices[0])
                    print(HumanNames)

You can see that no unknown class can be found.

Solution

You can try adding another threshold value and check if the predictions maximum value is lower than the threshold value. I have little experience in tensor flow so this is just a proof of concept, not sure if it will work.

best_class_probabilities = predictions[np.arange(len(best_class_indices)), best_class_indices] #original code
if(best_class_probabilities < threshold_2):
    best_class_indices = -1
HumanNames = "unknown"

By the way, because of the nature of triplet loss, you don't have to add and extra class to the SVC/SVM as the embedding model is locked and not trained, so unknown class embeddings will be very different to the known class. However you can try either approach.

Hope it can help you can have a nice day!