How does general image background removal AI work?

Question

I'm well aware of the inner workings of CNN models for object detection, and although I've not worked on a semantic segmentation problem I can imagine how it works.

With these types of models, we need to say "segment out the humans", or "segment out the X". But what about when I say something like "segment out the subject of this photo, whatever it happens to be". For example, see this service: https://removal.ai/

Without too much imagination I might guess that they apply a multiclass segmentation model and just show any foreground pixels, no matter what class they belong to. So we'd hope that the subject is in one of the classes that the model was trained for, and that there are no other class instances in the image that shouldn't be captured. But is there a more general way?

Answer 1

In image segmentation the target is actually an image, with the same dimensions as the input, where each pixel has a label depending on which class it represents. It is not uncommon for such a dataset to have a "background" class that essentially consists of the pixels not belonging to any other class. If not you can always group together classes typically associated with background (e.g. "sky", "cloud", "grass", "mountain", etc.) to form the class "background". Likewise you could group all other possible classes of interest (e.g. "person", "car", "horse", etc.) into the class "foreground". With this dataset you could train an image segmentation model that predicts if a pixel belongs to the background or the foreground, without actually classifying it into a "person" or a "car".

So suppose you want to make your own removal.ai, you could:

• find one or more diverse image segmentation datasets (it needs to be diverse so that it will work on any generic photo uploaded to the site)
• check all the unique classes in the labels
• group all classes associated with backgrounds into class 0 (i.e. "background" class)
• group all classes associated with foregrounds into class 1 (i.e. "foreground" class)
• train an image segmentation model with these two classes
• profit

Answer 2

Background removal is technically known as image matting. It is similar to segmentation, but it is a regression problem. The objective is to predict the alpha matte, which separates the foreground and background. Simply adding the predicted alpha matte as the fourth channel to the RGB image removes the background.

The model architecture is mostly an encoder-decoder model. The encoder extracts features and compresses them in the latent space, while the decoder constructs the alpha channel.

Since it is a challenging problem, most solutions require additional input along with the RGB image. The most common one is a trimap. The Deep Image Matting paper (Xu, N., Price, B., Cohen, S., & Huang, T., 2017) is one of the major studies and it proposes such a solution. However, trimap is a significant limitation since it requires human annotation.

To eliminate the trimap requirement, a model can be trained with only an RGB image, but the quality significantly decreases. Another solution is training two neural networks. The first one predicts a coarse alpha matte, and the second one returns a sharper alpha matte, as done in Google's Pixel 6 phones.

Another trimap-free solution is designing a model with one encoder and two decoders. The decoders predict a semantic mask and a detailed map simultaneously. GFM (Li, J., Zhang, J., Maybank, S. J., & Tao, D. (2020).) and MODNet (Ke, Z., Sun, J., Li, K., Yan, Q., & Lau, R. W. (2020).) are two related studies.

For more in-depth information on both traditional and deep learning solutions, you can refer to my blog post: https://withoutbg.com/resources/how-automatic-image-background-removal-works

Disclosure: I built withoutbg.com, a background removal tool, powered by deep learning.