iPhone X allows you to look at the TrueDepth camera and reports 52 facial blendshapes like how much your eye is opened, how much your jaw is opened, etc.

If I want to do something similar with other cameras (not TrueDepth), what are my alternative methods? Currently, I just use a simple ConvNet which takes in an image and predict 52 sigmoid values.

What do you think could be the underlying technology behind ARKit Face Tracking?


1 Answer 1


Facial classification and tracking is easier compared to tracking any other motion. This is due to the fact that the face have a large number of easily identifiable features.

Facial tracking is an additional layer on top of facial detection. Facial detection works by finding characteristics such as the cheekbones, chin, nose, eyes etc. These features are easy to detect as they have very specific properties. These points are found with the help of shadows and brightness, for example, the nose and cheekbones are highlighted, whereas the the eyes and lips have a shadows.

Using these feature points it is possible to create a mesh to construct the face. This has been a classical way to detect and classify faces. The TrueDepth camera helps to make a better structure by providing depth with the help of an IR sensor.

Facial features and Facial mesh

Facial tracking makes use of this mesh to study and track changes in the facial structure. For example when you smile, it will detect the cheekbone points changing, similarly when you open your mouth, your lips will move apart.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .