In a lot of explanations online for Xavier Initialization, I see the following:

With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a high value or vanishing to zero. In other words, we need to initialize the weights in such a way that the variance remains the same for x and y. This initialization process is known as Xavier initialization.

Source https://prateekvjoshi.com/2016/03/29/understanding-xavier-initialization-in-deep-neural-networks/

However, the intuition behind why var(output) should equal var(inputs) is never explained. Does anyone know why intuitively var(output) should equal var(inputs)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.