# How do we classify an unrecognised face in face recognition?

If we have classified 1000 people's faces; how do we ensure the network tells us when it encounters a new person?

## bumped to the homepage by Community♦yesterday

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• It's a hard problem, it's called "unknown class" and related to binary classification "one vs all". There is no established solution for it, only several heuristics. Simplest (but not good) is take threshold on probabilities of known classes. There are more complex solution with binary classifiers and hierarchies, but it's all on case-by-case basis. – mirror2image Mar 20 at 6:11

You can't really make the network tell you if a face is new (unless you actually train a network with that particular purpose, maybe). Ideally, if you feed a new face to a trained network, the output activations will be pretty low in all the possible categories (faces that the network has already seen) i.e. no particular category will signal a high probability. In that case, you can just set a cut, such that if the maximum activation is below a certain threshold, it means that the NN encountered a new face. But things are not that easy in practice. If the NN encounters a face it has already seen, but the image is weird (blurry, the face doesn't cover much of the actual image, there are multiple faces in the image etc.) you could get the same output as in the case of a new face. At the same time, for a NN images are just probability distributions, so it could happen that a new face, which for us, humans, looks nothing like the others, to look pretty similar to an old one, from the NN point of view, hence giving a high activation in a single particular output neuron.

Specifically for face recognition (and other identification algorithms) there are better approaches than using classifiers directly.

Most identity recognition algorithms generate some kind of metric - typically an "embedding" of the original image into an abstract space i.e. a vector of real numbers. The space might be based on real-world biometrics e.g. normalised measurements of eye distance, eyebrow arch etc, which would be trained as a regression algorithm. The problem with this is that it requires a lot of labelled data, and the biometrics are not necessarily good at differentiating between identities. An alternative is to get the neural network to find the best abstract space for identities. You can do this if you have at least two images of each identity, and using triplet loss to train the network - the loss function directly rewards embedding the same identity close and different identity far apart.

Once you have an embedding, you no longer directly classify identities using the neural network. Instead, you base identity on distance between measured embedding and stored embeddings. This requires implementing a search function that looks at known embeddings and sorts by distance.

how do we ensure the network tells us when it encounters a new person?

Embeddings don't solve this problem directly, but give a useful heuristic - distance in embedding space. Typically a maximum allowed distance is set as a cutoff to consider an image as showing a new identity. This is a hyperparameter of the model. This is an area that triplet loss helps with, since it is trained to make the distance as large as possible between images that show different identities. If it has generalised well during training, then it should ignore differences due to lighting, pose, makeup etc, but still be able to differentiate similar looking people.

As the embedding is approximate, any such system may make mistakes, and needs to be carefully tested. The quality and quantity of training data are important, and it should match images used in production. But that is no different to the pure classifier, which must in addition be re-built and re-trained for every new class added.

Whether to use a more basic classifier or something like triplet loss is a question of scale - if the number of identities that need to be tracked is high, or the rate of change in identities is high, then embeddings trained on triplet loss (or similar) are more practical.