# What is the intuition behind the Xavier initialization for deep neural networks?

The aim of weight initialization is to prevent layer activation outputs from exploding or vanishing during the course of a forward pass through a deep neural network

I am really having trouble understanding weights initialization technique and Xavier Initialization for deep neural networks (DNNs).

In simple words (and maybe with an example), what is the intuition behind the Xavier initialization for DNNs? When should we use Xavier's initialization?

The problem of exploding and vanishing gradients has been long-standing in the DL community. Initialize all the weights as zeros and the model learns identical features across all hidden layers, initialize random but large weights and the backpropagated gradients explode, initialize random but small weights and gradients vanish. The intuition is aptly captured by this simple mathematical observation: $$1.1^{50} = 117.390$$, while at the same time, $$0.9^{50} = 0.00515$$. Note that the difference between the two numbers is just $$0.1$$ but it has a tremendous effect when multiplied repeatedly! A typical NN is a series of function compositions involving weight matrices and linear/non-linear activation functions. When stripped to a bare minimum, it essentially is a series of matrix multiplications. Therefore, the way in which the elements of these weight matrices are initialized plays a major role in how the network learns.
• But even with Xavier initialization we can have vanishing gradients. For example in a FC with 10 layers and 1 neuron per layer, each neuron will be initialized from $\mathcal{N} (0, 1)$. These numbers can still be small, and as such their product $0.5^{10}$ can also be small. Jun 13 at 15:59