How does mixing and matching encoders and decoders work in image segmentation?

Question

I had a conceptual questions regarding architectures. I am using this git hub repository that allows one to quickly put together a segmentation pipeline. In reading the readme one thing that has me confused is separation of the encoder and decoders in the code base. I am using Unet which is listed as a decoder and I have resnet as my encoder.

This confuses me because the Unet paper states that it has both an encoder and decoder. Is it possible to simply mix and match as long as the layers are compatible. Lastly if I am using a pre-trained encoder with a UNet decoder am I only training half the model or just giving it a head start? I ask because I am able to do much larger image sizes with this splitting as oppose just having a pure Unet.

Thanks in advance!

Answer 1

It's possible to mix and match all sorts of encoders and decoders. If the output of the encoder can be mapped to the input of the decoder, and a loss function can be backpropagated through the model, then it is possible to combine them.

Image segmentation, however, can be done simply with U-NET, as it can be trained as an image segmentation model. You can use an encoder to 'encode' your image, to make it easier to segment with the U-NET. Im assuming that the Image-Segmentation library you linked is actually just doing that, using an encoder and then applying U-NET for image segmentation.