3
$\begingroup$

Two of the most popular initialization schemes for neural network weights today are Xavier and He. Both methods propose random weight initialization with a variance dependent on the number of input and output units. Xavier proposes

$$W \sim \mathcal{U}\Bigg[-\frac{\sqrt{6}}{\sqrt{n_{in}+n_{out}}},\frac{\sqrt{6}}{\sqrt{n_{in}+n_{out}}}\Bigg]$$

for networks with $\text{tanh}$ activation function and He proposes

$$W \sim \mathcal{N}(0,\sqrt{s/n_{in}})$$

for $\text{ReLU}$ activation. Both initialization schemes are implemented in the most commonly used deep learning libraries for python, PyTorch and TensorFlow.
However, for both versions we have a normal and uniform version. Now the main argument of both papers is about the variance of the information at initialization time (which is dependent on the non-linearity) and that it should stay constant across all layers when back-propagating. I see how one can simply adjust the bounds $[-a,a]$ of a uniform variable in such a way that the random variable has the desired standard deviation and vice versa ($\sigma = a/\sqrt{3}$), but I'm not sure why we need a normal and a uniform version for both schemes? Wouldn't it be just enough to have only normal or only uniform? Or uniform Xavier and normal He as proposed in their papers?

I can imagine uniform distributions are easier to sample from a computational point of view, but since we do the initialization operation only once at the beginning, the computational cost is negligible compared to that from training. Further uniform variables are bounded, so there are no long tail observations as one would expect in a normal. I suppose that's why both libraries have truncated normal initializations.

Are there any theoretical, computational or empirical justifications for when to use a normal over a uniform, or a uniform over a normal weight initialization regardless of the final weight variance?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.