Commonmark migration

I think they use the Xavier/Glorot's initialization method. You can read from [the original paper][1]the original paper:

We initialized the biases to be 0 and the weights $$W_{ij}$$ at each layer with the following commonly used heuristic:

$$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}]$$

where $$U[−a, a]$$ is the uniform distribution in the interval $$(−a, a)$$ and $$n$$ is the size of the previous layer (the number of columns of $$W$$)

Some people use this as some reports said this initialization method lead to better result [1]: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

malioboro
• 2.4k
• 2
• 11
• 36

I think they use the Xavier/Glorot's initialization method. You can read from [the original paper][1]:

We initialized the biases to be 0 and the weights $$W_{ij}$$ at each layer with the following commonly used heuristic:

$$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}]$$

where $$U[−a, a]$$ is the uniform distribution in the interval $$(−a, a)$$ and $$n$$ is the size of the previous layer (the number of columns of $$W$$)

Some people use this as some reports said this initialization method lead to better result [1]: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf