I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks.

The chain rule of differentiation for a composite function can be expressed roughly as follows:

$$\dfrac{d}{dx} (f_1(f_2(f_3(\cdots f_n(x))))) = \dfrac{df_n}{dx} \dfrac{df_{n-1}}{dy_1} \dfrac{df_{n-2}}{dy_2} \cdots \dfrac{df_1}{dy_{n-1}}$$

We know that multilayer perceptrons are composite functions of layer functions. So, if layers are increasing, then the gradient terms to multiply will increase on the right-hand side.

If all the gradient terms on the right-hand side are between 0 and 1 then the gradient will become less and less if layers keep on increasing the layers. This phenomenon is called the vanishing gradient problem. Similarly, if all the gradient terms on the right-hand side are greater than 1. Then the product will become more and more. This phenomenon is called exploding gradient problem.

Since it is customary to use the same activation function across all the layers in deep neural networks, all the gradients on the right hands behave in a similar manner, i.e. either most of the gradient terms on the right-hand side fall between 0 and 1 or greater than one, which causes either vanishing gradient or exploding gradient problem.

Is my mathematical interpretation of the vanishing and exploding gradient problem true? Am I missing anything?

  • 1
    $\begingroup$ It looks correct to me. At least, that's my understanding of vanishing/exploding gradients too. $\endgroup$
    – SpiderRico
    Sep 24 at 0:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.