1
$\begingroup$

I see many realtime face swap filters and appearance enhancement filters on smartphone apps. Even apps that can make you look like a granny or show you having a frown, no matter what your actual facial expression is.
On searching for open source algorithms/code to apply such effects to my desktop PC's webcam, all I find are hugely resource-heavy programs that require a dedicated GPU or TPU.
May I know how smartphone apps process face filters? If they send the data to a server for processing, how does it work in realtime? Or do they use scaled-down CNN models or GAN that can run on the phone's processor? If so, are there lightweight models or algorithms that can run on the desktop PC's CPU or integrated GPU (APU)?

$\endgroup$

1 Answer 1

2
$\begingroup$

Face filters works by first detecting and localizing the face, then predicting the so called facial landmarks (a set of points that depict the geometry of the face, like its contour, shape of eyes, nose, mouth, ecc), and lastly applying the filter potentially yield by some generative model.

These are all heavy work. For example, you can have a pipeline that first detects a face, if no faces you stop the processing. Otherwise you can have a multi-task model that yields the face bounding-box and landmarks directly, so improving the runtime because you reuse most of the model for both predictions. However, the part related for the filters is probably always separated, moreover you have to deal with computer graphics to render the filter and also apply some nice FX.

What I want to say is that both solutions are possible, and even a combination of them.

  • Many camera software (especially on mobile) have a very fast implementation of face detection (probably an improvement of Viola-Jones method). So once you detect a face, you can send that frame on the cloud for fast processing. Once you have the landmarks and, I guess, also the correct geometry of the filter, to enable real-time effects usually tracking is used. There exist very efficient tracking algorithms that rely un classical computer vision, that can either track objects and even landmarks. Basically you run the expensive part (or cloud processing) one frame every $N$, and for the other you use only tracking.
  • Scaled-down CNN models or GAN, are also possible. There exist optimization frameworks like tf-lite that allow you to compress (by quantization and pruning), optimize, and even deploy a DL model. Indeed, you need to care about specific optimizations for the CPU and GPU.

If you plan to implement something similar on your on PC, you can use OpenCV (either with Python or C++) for, e.g., tracking, then compress a DL model with tf-lite in case you use tensorflow and/or Keras: you can take a pre-trained model, and compress with such tool. In that way, you can target optimized CPU and GPU models. For the graphics part you should find some optimized library for GPU, or write the code yourself like in OpenGL or similar. You can also have a look at Dlib: it's a nice lib, that has many image processing, computer vision, ML implementations, and even pre-trained models.

$\endgroup$
3
  • $\begingroup$ Thank you Luca. By "one frame every N", I assume you mean "one frame every N number of seconds"? So the tracking algorithms run on the phone to track facial features. Any such algorithm names you remember? $\endgroup$
    – Julian
    Commented May 31, 2023 at 14:47
  • $\begingroup$ To achieve real-time processing you should process at least 24 image frames in a second (say up to 60 for best smoothness). It's not necessary to either gather more of them, so you should capture camera images on a fixed interval basis. Assuming that, you do all the heavy work at frame 1, then for the next $N$ frames (e.g. 10) you do tracking ($N$ should be not too large otherwise you loose accuracy). At frame $N+1$ you do the heavy work again, and so on. For tracking you can start from the Lucas-Kanade method. $\endgroup$ Commented May 31, 2023 at 17:03
  • $\begingroup$ That's great! Thank you. $\endgroup$
    – Julian
    Commented Jun 2, 2023 at 2:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .