# What is the intuition behind the Xavier initialization for deep neural networks?

The aim of weight initialization is to prevent layer activation outputs from exploding or vanishing during the course of a forward pass through a deep neural network

I am really having trouble understanding weights initialization technique and Xavier Initialization for deep neural networks (DNNs).

In simple words (and maybe with an example), what is the intuition behind the Xavier initialization for DNNs? When should we use Xavier's initialization?

The problem of exploding and vanishing gradients has been long-standing in the DL community. Initialize all the weights as zeros and the model learns identical features across all hidden layers, initialize random but large weights and the backpropagated gradients explode, initialize random but small weights and gradients vanish. The intuition is aptly captured by this simple mathematical observation: $$1.1^{50} = 117.390$$, while at the same time, $$0.9^{50} = 0.00515$$. Note that the difference between the two numbers is just $$0.1$$ but it has a tremendous effect when multiplied repeatedly! A typical NN is a series of function compositions involving weight matrices and linear/non-linear activation functions. When stripped to a bare minimum, it essentially is a series of matrix multiplications. Therefore, the way in which the elements of these weight matrices are initialized plays a major role in how the network learns.