How are exploding numbers in a forward pass of a CNN combated?

Question

Take AlexNet for example:

In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example:

Say I have a weight matrix of [-1,-2,3,4] and inputs of [ReLU(4), ReLU(5), ReLU(-2), Relu(-3)]. The resultant matrix from these will have large numbers for the inputs of ReLU(4) and ReLU(5), and 0 for ReLU(-2) and ReLU(-3). If there are even just a few more layers, the numbers are quick to either explode or be 0.

How is this typically combated? How do you keep these numbers towards 0? I understand you can take subtract the mean at the end of each layer, but for a layer that is already in the millions, subtracting the mean will still result in thousands.

Answer 1

The most effective way to prevent both the forward and backward propagation of exploding is keeping the weights in a small range. The main way this is accomplished is through their initialization.

For example in the case of He initialization, the authors show (given some assumptions) that the variance of the output of the final layer $L$ of the network is:

$$ Var[y_L] = Var[y_1] \left( \prod_{i=2}^L{\frac{1}{2} \, n_l \, Var[w_l]} \right) $$

where $n_l$ and $w_l$ are the number of connections and weights of layer $l$. In order to keep the outputs from exploding the product above should not exponentially magnify its inputs. In order to do this the authors elect to initialize the weights so that:

$$ \frac{1}{2} \, n_l \, Var[w_l] = 1 $$

Now this is helps keeping the outputs from exploding. They then go and prove that the same strategy helps preventing the gradients from exploding.

Another similar strategy is the so-called Glorot (or Xavier) Initialization. These techniques are extremely effective in helping the models converge!