How does general image background removal AI work?

Question

I'm well aware of the inner workings of CNN models for object detection, and although I've not worked on a semantic segmentation problem I can imagine how it works.

With these types of models, we need to say "segment out the humans", or "segment out the X". But what about when I say something like "segment out the subject of this photo, whatever it happens to be". For example, see this service: https://removal.ai/

Without too much imagination I might guess that they apply a multiclass segmentation model and just show any foreground pixels, no matter what class they belong to. So we'd hope that the subject is in one of the classes that the model was trained for, and that there are no other class instances in the image that shouldn't be captured. But is there a more general way?

Answer 1

In image segmentation the target is actually an image, with the same dimensions as the input, where each pixel has a label depending on which class it represents. It is not uncommon for such a dataset to have a "background" class that essentially consists of the pixels not belonging to any other class. If not you can always group together classes typically associated with background (e.g. "sky", "cloud", "grass", "mountain", etc.) to form the class "background". Likewise you could group all other possible classes of interest (e.g. "person", "car", "horse", etc.) into the class "foreground". With this dataset you could train an image segmentation model that predicts if a pixel belongs to the background or the foreground, without actually classifying it into a "person" or a "car".

So suppose you want to make your own removal.ai, you could:

• find one or more diverse image segmentation datasets (it needs to be diverse so that it will work on any generic photo uploaded to the site)
• check all the unique classes in the labels
• group all classes associated with backgrounds into class 0 (i.e. "background" class)
• group all classes associated with foregrounds into class 1 (i.e. "foreground" class)
• train an image segmentation model with these two classes
• profit