0
$\begingroup$

Is the only difference between the two how the skip connection is combined? Resnet combines skip connections through addition and Densenet through concatenating.

The Densenet paper appears to be arguing that their approach can allow gradients and information to flow from any previous layer to any future layer. However, that doesn't make sense since Resnet can already do that.

$\endgroup$

2 Answers 2

1
$\begingroup$

ResNet on the left side, DenseNet on the right side.

The advantages that are cited in the paper are:

  • alleviate the vanishing gradient problem -- which is true. The purpose of the ResNet is NOT to solve the vanishing gradient problem, but answering the question how to make deeper models actually better approximators?
  • strengthen feature propagation and encourage feature reuse -- which is also true. You can see that in the DenseNet the raw features from the first layer are propagated forwards to every next layer. While in the ResNet the features of the first layer are mixed with the next through addition and non-linearity, and so on..
  • reduce the number of parameters -- which is also true. This sounds a bit counter-intuitive but the fact is that these types of layers require less filter maps than the ResNet, resulting in fewer parameters. For a comparison you can checkout an experiment that I did on CIFAR-10 here:
    https://github.com/pi-tau/deep-conv-nets?tab=readme-ov-file#model-comparison-on-cifar10

ResNet vs DenseNet

$\endgroup$
0
$\begingroup$

DenseNet and ResNet are both CNN architectures that effectively use skip connections in order to facilitate a better flow of gradients throughout the network.

Coming to the differences,

Implementation of Skip Connections

You are correct in saying that ResNet combines skip connections through additive identity transformations, whereas DenseNet uses concatenation to combine the outputs of each layer with feature maps of all preceding layers. The reason why I highlighted all preceding layers is that it is a key feature that distinguishes DenseNet based architectures from ResNet based architectures.

WHY?

A typical ResNet will have skip connections that will skip some layers and will allow the network's current layer to perform atleast as good as the previous layers. DenseNet on the other hand utilizes the feature maps created by all preceding layers which enables feature reuse which leads to better feature representation.

Parameters v/s Memory

The number of parameters present in a ResNet are substantially large because each layer has its own weights. In comparision, DenseNets add only a small subset of feature maps (at the current layer) to the previously existing "knowledge" i.e. the feature maps provided by the previous layers which makes it parameter efficient.

However, since DenseNet is concatenating the feature maps, as the network becomes deep, the number of feature maps increase which results in an increased demand in memory.

That being said, DenseNets are known to achieve comparable or better results with fewer number of parameters than ResNets.

The Densenet paper appears to be arguging that their approach can allow gradients and information to flow from any previous layer to any future layer. However that doesn't make sense since Resnet can already do that.

The gap in your understanding here lies in the fact that the DenseNet paper allows the gradients to flow from all previous layers to the current layer.

Remember - Denser the connectivity of your skip connections, easier the gradient flow will be throughout the network. (Till a certain level, after which it might start hindering the performance. As always, take every advice with a pinch of salt, there is no free lunch :)

Update

Let me present it in a more mathematical manner,

A typical convolution, as we know, can be formulated as follows: $$f_{l} = H_{l}*f_{l-1}$$ where $f_{l}$ is the current feature map, $f_{l-1}$ is the previous feature map, and $*$ is the convolution operator. $H_{l}$ signifies the weight of the convolution.

Note - We are not considering the bias for ease of understanding.

Now, in a DenseNet, $f_{l-1}$ is connected to all preceding convolutional blocks,

$$Y_{l} = X_{0}.X_{1}.\cdots.X_{l}$$ where $X_{i}$ represents the feature maps of all preceding layers and $.$ represents the concatenation operation.

Replacing $f_{l-1}$ with $Y_{l}$, we get,

$$f_{l} = H_{l} * (X_{0}.X_{1}.\cdots.X_{l})$$

If we divide the weight $H_{l}$ into multiple smaller weights, we get,

$$f_{t} = (H_{l}^{0}.H_{l}^{1}.\cdots.H_{l}^{l})*(X_{0}.X_{1}.\cdots.X_{l})$$ $$=H_{l}^{0}*X_{0} + H_{l}^{1}*X_{1} + \cdots +H_{l}^{l}*X_{l}$$

For ResNet, since the connections are added together, it is trivial to show the following,

$$f_{t} = H_{l}*(X_{0}+X_{1}+\cdots+X_{l})$$ $$=H_{l}*X_{0} + H_{l}*X_{1} + \cdots +H_{l}*X_{l}$$

Now, we can see that for ResNet, we are using the same convolutional weight for all the previous layers, while the weights are different for DenseNet. Therefore, DenseNet has more flexibility as to how much weight should be given to any of the previous layer features whereas in the case of ResNet, the current layer determines the weight for all the features of the preceding layer.

This results in DenseNet's ability to use all the features of the preceding layers as is i.e., without changing the weights again and again.

Hope this makes it clearer to understand.

$\endgroup$
4
  • $\begingroup$ It's not special that densenet takes in gradients/features from all previous layers because resnet does the same, as per my last sentence $\endgroup$ Mar 7 at 21:05
  • $\begingroup$ You should really look up how ResNet works. The skip connections in a ResNet-based architecture do not account for all of the previous features; they simply add the input of a layer to the output of another layer after skipping one or more layers in between. So, while the current layer might receive inputs from earlier layers, it does not necessarily imply that all the previous features will be incorporated into the output layer. $\endgroup$ Mar 8 at 13:44
  • $\begingroup$ You're wrong about this. It does imply that all previous features are incorporated into the output layer. $\endgroup$ Mar 8 at 20:29
  • $\begingroup$ Does the edit answer your question? $\endgroup$ Mar 9 at 17:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .