Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?

Question

I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks.

The chain rule of differentiation for a composite function can be expressed roughly as follows:

$$\dfrac{d}{dx} (f_1(f_2(f_3(\cdots f_n(x))))) = \dfrac{df_n}{dx} \dfrac{df_{n-1}}{dy_1} \dfrac{df_{n-2}}{dy_2} \cdots \dfrac{df_1}{dy_{n-1}}$$

We know that multilayer perceptrons are composite functions of layer functions. So, if layers are increasing, then the gradient terms to multiply will increase on the right-hand side.

If all the gradient terms on the right-hand side are between 0 and 1 then the gradient will become less and less if layers keep on increasing the layers. This phenomenon is called the vanishing gradient problem. Similarly, if all the gradient terms on the right-hand side are greater than 1. Then the product will become more and more. This phenomenon is called exploding gradient problem.

Since it is customary to use the same activation function across all the layers in deep neural networks, all the gradients on the right hand behave in a similar manner, i.e. either most of the gradient terms on the right-hand side fall between 0 and 1 or greater than one, which causes either vanishing gradient or exploding gradient problem.

Do my mathematical interpretation of the vanishing and exploding gradient problem true? Am I missing anything?

Answer 1

Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can visualize them here):

1. If the terms are in $(-1,1)$, in their limit they tend to 0.
2. If they are all 1, they stay at 1.
3. If they are all -1, they alternate between -1 and 1.
4. If they are greater than 1, they tend to $\infty$. If they are smaller than -1 they tend to $-\infty$.

In practice, however, cases 1. and 4. are the most common, and most of the strategies (e.g. resnet, lstm) are designed to tackle this problem as you probably well know already.