Background removal is technically known as image matting. It is similar to segmentation, but it is a regression problem. The objective is to predict the alpha matte, which separates the foreground and background. Simply adding the predicted alpha matte as the fourth channel to the RGB image removes the background.
The model architecture is mostly an encoder-decoder model. The encoder extracts features and compresses them in the latent space, while the decoder constructs the alpha channel.
Since it is a challenging problem, most solutions require additional input along with the RGB image. The most common one is a trimap. The Deep Image Matting paper (Xu, N., Price, B., Cohen, S., & Huang, T., 2017) is one of the major studies and it proposes such a solution. However, trimap is a significant limitation since it requires human annotation.
To eliminate the trimap requirement, a model can be trained with only an RGB image, but the quality significantly decreases. Another solution is training two neural networks. The first one predicts a coarse alpha matte, and the second one returns a sharper alpha matte, as done in Google's Pixel 6 phones.
Another trimap-free solution is designing a model with one encoder and two decoders. The decoders predict a semantic mask and a detailed map simultaneously. GFM (Li, J., Zhang, J., Maybank, S. J., & Tao, D. (2020).) and MODNet (Ke, Z., Sun, J., Li, K., Yan, Q., & Lau, R. W. (2020).) are two related studies.
For more in-depth information on both traditional and deep learning solutions, you can refer to my blog post: https://withoutbg.com/resources/how-automatic-image-background-removal-works
Disclosure: I built withoutbg.com, a background removal tool, powered by deep learning.