Take AlexNet for example:

combined gpu stream of alexnet

In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example:

Say I have a weight matrix of [-1,-2,3,4] and inputs of [ReLU(4), ReLU(5), ReLU(-2), Relu(-3)]. The resultant matrix from these will have large numbers for the inputs of ReLU(4) and ReLU(5), and 0 for ReLU(-2) and ReLU(-3). If there are even just a few more layers, the numbers are quick to either explode or be 0.

How is this typically combated? How do you keep these numbers towards 0? I understand you can take subtract the mean at the end of each layer, but for a layer that is already in the millions, subtracting the mean will still result in thousands.


The most effective way to prevent both the forward and backward propagation of exploding is keeping the weights in a small range. The main way this is accomplished is through their initialization.

For example in the case of He initialization, the authors show (given some assumptions) that the variance of the output of the final layer $L$ of the network is:

$$ Var[y_L] = Var[y_1] \left( \prod_{i=2}^L{\frac{1}{2} \, n_l \, Var[w_l]} \right) $$

where $n_l$ and $w_l$ are the number of connections and weights of layer $l$. In order to keep the outputs from exploding the product above should not exponentially magnify its inputs. In order to do this the authors elect to initialize the weights so that:

$$ \frac{1}{2} \, n_l \, Var[w_l] = 1 $$

Now this is helps keeping the outputs from exploding. They then go and prove that the same strategy helps preventing the gradients from exploding.

Another similar strategy is the so-called Glorot (or Xavier) Initialization. These techniques are extremely effective in helping the models converge!

| improve this answer | |
  • $\begingroup$ I had thought maybe this could be a solution, but assumed there was a way of applying this such that it was independent of the number of weights/layers. Thankyou for linking the paper, I'll give it a read when I get the time $\endgroup$ – Recessive Jun 29 '19 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.