Progress in many application tasks in artificial intelligence is achieved by increasing the depth of the neural networks. But if we keep on increasing the number of layers in the neural network, the performance of the neural network saturates and then generally declines.
I know only the following reasons for such a decline: Vanishing or exploding gradients. This can be fixed, at-least to some extent, by using normalisation techniques.
What are all the known possible reasons for such a decline mint in the performance of the neural network?