It's possible to mix and match all sorts of encoders and decoders. If the output of the encoder can be mapped to the input of the decoder, and a loss function can be backpropagated through the model, then it is possible to combine them.

However, although many encoders go with many decoders, it does not necessarily mean that the performance will be any good. For your example, in particular, part of the appeal of U-NET is the skip connections. If you remove the U-NET encoder and use a different encoder, then those skip connections will not exist, and I would assume the resulting model will not perform incredibly well.

It's possible to mix and match all sorts of encoders and decoders. If the output of the encoder can be mapped to the input of the decoder, and a loss function can be backpropagated through the model, then it is possible to combine them.

Image segmentation, however, can be done simply with U-NET, as it can be trained as an image segmentation model. You can use an encoder to 'encode' your image, to make it easier to segment with the U-NET. Im assuming that the Image-Segmentation library you linked is actually just doing that, using an encoder and then applying U-NET for image segmentation.