Progress in many application tasks in artificial intelligence is achieved by increasing the depth of the neural networks. But if we keep on increasing the number of layers in the neural network, the performance of the neural network saturates and then generally declines.

I know only the following reasons for such a decline: Vanishing or exploding gradients. This can be fixed, at-least to some extent, by using normalisation techniques.

What are all the known possible reasons for such a decline mint in the performance of the neural network?

  • 1
    $\begingroup$ A related question in case someone might be able to address in their answers: What are the gaps and remaining difficulties in dealing with the vanishing gradient problem, when fitting a deep network? E.g. LSTM still does not do well in long sentences $\endgroup$
    – siegfried
    Sep 22 at 2:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.