I had a conceptual questions regarding architectures. I am using this git hub repository that allows one to quickly put together a segmentation pipeline. In reading the readme one thing that has me confused is separation of the encoder and decoders in the code base. I am using Unet which is listed as a decoder and I have resnet as my encoder.

This confuses me because the Unet paper states that it has both an encoder and decoder. Is it possible to simply mix and match as long as the layers are compatible. Lastly if I am using a pre-trained encoder with a UNet decoder am I only training half the model or just giving it a head start? I ask because I am able to do much larger image sizes with this splitting as oppose just having a pure Unet.

Thanks in advance!

  • $\begingroup$ welcome to this SE :) $\endgroup$ Feb 22, 2023 at 9:25
  • $\begingroup$ Ive never heard of projects splitting up a U-NET, because it kind of ruins the idea of U-NET. Are you sure you are using only the decoder part? And not just the complete U-NET architecture? $\endgroup$ Feb 22, 2023 at 9:26
  • $\begingroup$ @RobinvanHoorn per the git hub unless I am misunderstanding what they are doing they are only using the UNet architecture for the decoding. That package actually does not let you train a model from scratch far as I am aware. I am new so I could be misreading. $\endgroup$ Feb 22, 2023 at 16:28

1 Answer 1


It's possible to mix and match all sorts of encoders and decoders. If the output of the encoder can be mapped to the input of the decoder, and a loss function can be backpropagated through the model, then it is possible to combine them.

Image segmentation, however, can be done simply with U-NET, as it can be trained as an image segmentation model. You can use an encoder to 'encode' your image, to make it easier to segment with the U-NET. Im assuming that the Image-Segmentation library you linked is actually just doing that, using an encoder and then applying U-NET for image segmentation.

  • $\begingroup$ Thanks for clearing that up. $\endgroup$ Feb 22, 2023 at 16:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .