I have seen in several jupyter notebooks people initializing the NN weights using:

np.random.randn(D, M) / np.sqrt(D)

Other times they just do:

np.random.randn(D, M)

What is the advantage of dividing the Gaussian distribution by the squared root of the number of neurons in the layer?



I think they use the Xavier/Glorot's initialization method. You can read from the original paper:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:

$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $

where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.