The question is about the architecture of Deep Residual Networks (ResNets). The model that won the 1-st places at "Large Scale Visual Recognition Challenge 2015" (ILSVRC2015) in all five main tracks:
- ImageNet Classification: “Ultra-deep” (quote Yann) 152-layer nets
- ImageNet Detection: 16% better than 2nd
- ImageNet Localization: 27% better than 2nd
- COCO Detection: 11% better than 2nd
- COCO Segmentation: 12% better than 2nd
Source: MSRA @ ILSVRC & COCO 2015 competitions (presentation, 2-nd slide)
This work is described in the following article:
Microsoft Research team (developers of ResNets: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun) in their article:
state that depth plays a key role:
"We obtain these results via a simple but essential concept — going deeper. These results demonstrate the potential of pushing the limits of depth."
It is emphasized in their presentation also (deeper - better):
- "A deeper model should not have higher training error."
- "Deeper ResNets have lower training error, and also lower test error."
- "Deeper ResNets have lower error."
- "All benefit more from deeper features – cumulative gains!"
- "Deeper is still better."
But recently I have found one theory that introduces a novel interpretation of residual networks showing they are exponential ensembles:
Deep Resnets are described as many shallow networks whose outputs are pooled at various depths. There is a picture in the article. I attach it with explanation:
Residual Networks are conventionally shown as (a), which is a natural representation of Equation (1). When we expand this formulation to Equation (6), we obtain an unraveled view of a 3-block residual network (b). From this view, it is apparent that residual networks have O(2^n) implicit paths connecting input and output and that adding a block doubles the number of paths.
In conclusion of the article it is stated:
It is not depth, but the ensemble that makes residual networks strong. Residual networks push the limits of network multiplicity, not network depth. Our proposed unraveled view and the lesion study show that residual networks are an implicit ensemble of exponentially many networks. If most of the paths that contribute gradient are very short compared to the overall depth of the network, increased depth alone can’t be the key characteristic of residual networks. We now believe that multiplicity, the network’s expressability in the terms of the number of paths, plays a key role.
But it is only a recent theory that can be confirmed or refuted. It happens sometimes that some theories are refuted and articles are withdrawn.
Should we think of deep ResNets as an ensemble after all? Ensemble or depth makes residual networks so strong? Is it possible that even the developers themselves do not quite perceive what their own model represents and what is the key concept in it?