Take AlexNet for example:
In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example:
Say I have a weight matrix of
[-1,-2,3,4] and inputs of
[ReLU(4), ReLU(5), ReLU(-2), Relu(-3)]. The resultant matrix from these will have large numbers for the inputs of
ReLU(5), and 0 for
ReLU(-3). If there are even just a few more layers, the numbers are quick to either explode or be 0.
How is this typically combated? How do you keep these numbers towards 0? I understand you can take subtract the mean at the end of each layer, but for a layer that is already in the millions, subtracting the mean will still result in thousands.