# Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer?

I have seen in several jupyter notebooks people initializing the NN weights using:

np.random.randn(D, M) / np.sqrt(D)


Other times they just do:

np.random.randn(D, M)


What is the advantage of dividing the Gaussian distribution by the squared root of the number of neurons in the layer?

Thanks

We initialized the biases to be 0 and the weights $$W_{ij}$$ at each layer with the following commonly used heuristic:
$$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}]$$
where $$U[−a, a]$$ is the uniform distribution in the interval $$(−a, a)$$ and $$n$$ is the size of the previous layer (the number of columns of $$W$$)