Should deep residual networks be viewed as an ensemble of networks?

Question

The question is about the architecture of Deep Residual Networks (ResNets). The model that won the 1-st places at "Large Scale Visual Recognition Challenge 2015" (ILSVRC2015) in all five main tracks:

• ImageNet Classification: “Ultra-deep” (quote Yann) 152-layer nets
• ImageNet Detection: 16% better than 2nd
• ImageNet Localization: 27% better than 2nd
• COCO Detection: 11% better than 2nd
• COCO Segmentation: 12% better than 2nd
  
  Source: MSRA @ ILSVRC & COCO 2015 competitions (presentation, 2-nd slide)

This work is described in the following article:

Deep Residual Learning for Image Recognition (2015, PDF)

---

Microsoft Research team (developers of ResNets: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun) in their article:

"Identity Mappings in Deep Residual Networks (2016)"

state that depth plays a key role:

"We obtain these results via a simple but essential concept — going deeper. These results demonstrate the potential of pushing the limits of depth."

It is emphasized in their presentation also (deeper - better):

- "A deeper model should not have higher training error."
- "Deeper ResNets have lower training error, and also lower test error."
- "Deeper ResNets have lower error."
- "All benefit more from deeper features – cumulative gains!"
- "Deeper is still better."

Here is the sctructure of 34-layer residual (for reference):

---

But recently I have found one theory that introduces a novel interpretation of residual networks showing they are exponential ensembles:

Residual Networks are Exponential Ensembles of Relatively Shallow Networks (2016)

Deep Resnets are described as many shallow networks whose outputs are pooled at various depths. There is a picture in the article. I attach it with explanation:

Residual Networks are conventionally shown as (a), which is a natural representation of Equation (1). When we expand this formulation to Equation (6), we obtain an unraveled view of a 3-block residual network (b). From this view, it is apparent that residual networks have O(2^n) implicit paths connecting input and output and that adding a block doubles the number of paths.

In conclusion of the article it is stated:

It is not depth, but the ensemble that makes residual networks strong. Residual networks push the limits of network multiplicity, not network depth. Our proposed unraveled view and the lesion study show that residual networks are an implicit ensemble of exponentially many networks. If most of the paths that contribute gradient are very short compared to the overall depth of the network, increased depth alone can’t be the key characteristic of residual networks. We now believe that multiplicity, the network’s expressability in the terms of the number of paths, plays a key role.

But it is only a recent theory that can be confirmed or refuted. It happens sometimes that some theories are refuted and articles are withdrawn.

---

Should we think of deep ResNets as an ensemble after all? Ensemble or depth makes residual networks so strong? Is it possible that even the developers themselves do not quite perceive what their own model represents and what is the key concept in it?

Answer 1

Imagine a genie grants you three wishes. Because you are an ambitious deep learning researcher your first wish is a perfect solution for a 1000-layer NN for Image Net, which promptly appears on your laptop.

Now a genie induced solution doesn't give you any intuition how it might be interpreted as an ensemble, but do you really believe that you need 1000 layers of abstraction to distinguish a cat from a dog? As the authors of the "ensemble paper" mention themselves, this is definitely not true for biological systems.

Of course you could waste your second wish on a decomposition of the solution into an ensemble of networks, and I'm pretty sure the genie would be able to oblige. The reason being that part of the power of a deep network will always come from the ensemble effect.

So it is not surprising that two very successful tricks to train deep networks, dropout and residual networks, have an immediate interpretation as implicit ensemble. Therefore "it's not depth, but the ensemble" strikes me as a false dichotomy. You would really only say that if you honestly believed that you need hundreds or thousands of levels of abstraction to classify images with human accuracy.

I suggest you use the last wish for something else, maybe a pinacolada.

Answer 2

Random residual networks for many non-linearities such as tanh live on the edge of chaos, in that the cosine distance of two input vectors will converge to a fixed point at a polynomial rate, rather than an exponential rate, as with vanilla tanh networks. Thus a typical residual network will slowly cross the stable-chaotic boundary with depth, hovering around this boundary for many layers. Basically it does not “forget” the geometry of the input space “very quickly”. So even if we make them considerably deep, they work better the vanilla networks.

For more information on the propagation of information in residual networks - Mean Field Residual Networks: On the Edge of Chaos