I was able to find that the skip connections used in U-Net help to recover fine grained details in the prediction, however I do not understand what is meant by this. Besides, I was wondering what would happen if the U-Net does not include skip connections.
1 Answer
$\begingroup$
$\endgroup$
You should checkout the original resnet paper which popularized skip connections, or shortcut connections, in the modern literature. See Related Work Section 2 in the paper.
Basically though, residiual connections help "remind" the network of what it was trying to learn initialy. It's like if you were to give a presentation. You always start a presentation giving the general outline (the entire image) go into details on a few specific topics (deep convolutions) and then remind the audience of where they started so they can tie everything together. Sometimes the audience doesn't have that "ah-ha!" moment until they see both the big picture and the smaller details.
Without the skip connections, there may be a loss of information from layer to layer. It's possible that the output of the net must be a function of both the input image and some very fine-grained deep features. Without skip connections, this would not be possible. A concrete example could be that in order to detect an emergency vehicle, you need to detect both the larger feature (the vehicle) and the more fine-grained feature (the siren) and combine them later in the network.
For UNet specfically, it was built for images of varying sizes. So the larger and smaller convolutional sizes help to extract features at various resolutions. Plus, if you didnt have skip-layres then when trying to upsample the image back to the original resolution it might be really difficult to decode the image as it did before. Unet was built originally for very accurate image segmentation, meaning that it needed to segment an image to pixel-perfect accuracy if it could. Make the net decode the exact line features would be very difficult through the various layers of convolutions, and even unnecessary for this task.
Granted, my explanation was pretty off-the-cuff. This article by Nikolas Adaloglou does a decent job,
"To sum up, the motivation behind this type of skip connections is that they have an uninterrupted gradient flow from the first layer to the last layer, which tackles the vanishing gradient problem. Concatenative skip connections enable an alternative way to ensure feature reusability of the same dimensionality from the earlier layers and are widely used.
On the other hand, long skip connections are used to pass features from the encoder path to the decoder path in order to recover spatial information lost during downsampling. Short skip connections appear to stabilize gradient updates in deep architectures. Finally, skip connections enable feature reusability and stabilize training and convergence."
